Friday, December 27, 2019

Python MRO (Method Resolution Order)

The python MRO defines the class search path used by Python to search for the right method to use in classe having multi-inheritance. Python supports class inheriting from other classes, the class being inheritated is called PARENT or SUPER class, while the class that inherits is called CHILD or SUB class. In python method resolution order defines the order in which the base classes are searched when executing a method. This order is also called LINEARIZATION of a class and set of rules are called MRO(Method Resolution Order).

For Example:


class A:
    def callme(self):
        print("From class A")

class B(A):
    def callme(self):
        print("From class B")

class C(A, B):
    def callme(self):
        print("From class C")

c = C()
c.callme()

As shown in above example when "callme" method executed on class C instance Python MRO decides which methods to be executed.
What is solution to the above problem ?
Solution to above problem is Two Algorithms to find Method Resolution Order.

                    Solution
                       |
                       |
                 Two Algorithms
                       |
                      / \
                    /     \
                  /         \
              Old MRO    New MRO (C3 Algorithm)
                 |           |
            Old Classes  New Classes

Old Class and New Class Example:

Old Classes:
  • Doesn't inherit from python root object class.
  • Uses old python MRO algorithm (DLR)

class A:
    def callme(self):
        pass

New Classes
  • First Parent inherit from python root object class.
  • Uses new python MRo algorithm (C3).
  • Introduced in python 2.3
class A(object):
    def callme(self):
        pass

History of Python MRO


Everything started in October 2002 with a post by Samuele Pedroni to the Python development mailing list. In his post, Samuele showed that the Python 2.2 method resolution order is not monotonic and he proposed to replace it with the C3 method resolution order. Guido agreed with his arguments and therefore now Python 2.3 uses C3. The C3 method itself has nothing to do with Python, since it was invented by people working on Dylan and it is described in a paper intended for lispers. The present paper gives a (hopefully) readable discussion of the C3 algorithm for Pythonistas who want to understand the reasons for the change.
  • Here is the link with detailed explaination of MRO introduced in python 2.3.
  • Here is the link to Samuele Pedroni's email to python mailing list.

What is Old MRO algorithm ?

Python old mro uses DLR algorithm i.e. Depth First, Left to Right. During implementing multiple inheritances, Python builds a list of classes to search as it needs to resolve which method has to be called when one is invoked by an instance. As the name suggests, the method resolution order will search the depth-first, then go left to right.
class A:
    pass

class B:
    pass

class C(A, B):
    pass

class D(B, A):
    pass

class E(C,D):
    pass

In the above Example algorithm first looks into the instance class for the invoked method. If not present, then it looks into the first parent, if that too is not present then-parent of the parent is looked into. This continues till the end of the depth of class and finally, till the end of inherited classes. So, the resolution order in our last example will be D, B, A, C, A. But, A cannot be twice present thus, the order will be D, B, A, C. But this algorithm varying in different ways and showing different behaviours at different times.

So, why Python search for A before C when A does not inherits from object in Python 2 ? This is due to the old classes MRO algorithm behaviour. It is a very simple and easy to understand algorithm : When a class inherits from multiple parents, Python build a list of classes to search for when it needs to resolve which method has to be called when one in invoked by an instance. This algorithm is a tree routing, and works this way, deep first, from left to right :
  1. Look if method exists in instance class
  2. If not, looks if it exists in its first parent, then in the parent of the parent and so on
  3. If not, it looks if current class inherits from others classes up to the current instance others parents.
So in our example, algorithm search path is : D, B, A, C, A. A class cannot appears twice in search path, so the final version is D, B, A, C:
Looking in D If not found, looking in B If not found, looking un B first parent A If not found, going back in B others parents (none) If not found, looking in D others parents : C

What is New MRO algorithm ?


C3 Linearization algorithm is an algorithm that uses new-style classes. It is used to remove an inconsistency created by DLR Algorithm. It has certain limitation they are: Children precede their parents. If a class inherits from multiple classes, they are kept in the order specified in the tuple of the base class.

C3 Linearization Algorithm works on three rules: Inheritance graph determines the structure of method resolution order. User have to visit the super class only after the method of the local classes are visited. Monotonicity.

As shown in example of new classes, when the class inherits from Python root object the behaviour changes. If you run the above using python3 the algorithm behaviour change too. This is because in Python 3 or in Python 2 using object as a parent class the new MRO algorithm is used.
So, what happens if we run above example using python 3 ?


class A(object):
    pass

class B(object):
    pass

class C(A, B):
    pass

class D(B, A):
    pass

class E(C,D):
    pass

TypeError: Error when calling the metaclass bases Cannot create a consistent method resolution order (MRO) for bases A, B

You will see above error when you run above code with python 3. Python 3 MRO algorithm is not able to find proper method resolution for above inherited classes so it breaks and throws typeerror related to mro.
Lets modify above code to work with python 3 and find out what is mro for above code.


class A(object):
    pass

class B(object):
    pass

class C(A, B):
    pass

class D(A, B):
    pass

class E(C,D):
    pass

E.__mro__

(<class '__main__.E'>, <class '__main__.C'>, <class '__main__.D'>, <class '__main__.A'>, <class '__main__.B'>, <type 'object'>)


As shown in above output MRO for given code is E, C, D, A, B using new c3 linerization algorithm.

References

Tuesday, December 17, 2019

Python Copy and Deepcopy


Assignment statements (=) in Python do not copy objects, they create bindings between a target and an object. When user use = operator user thinks that this creates a new object well, it doesn’t. It only creates a new variable that shares the reference of the original object. Sometimes a user wants to work with mutable objects, in order to do that user looks for a way to create “real copies” or “clones” of these objects. Or, sometimes a user wants copies that user can modify without automatically modifying the original at the same time, in order to do that we create copies of objects.


In Python, there are two ways to create copies:

*   Deep copy
*   Shallow copy

Above two methods are provided by python "copy" module.

What is Shallow Copy ?


A shallow copy means constructing a new collection object and then populating it with references to the child objects found in the original. In case of shallow copy, a reference of object is copied in other object. It means that any changes made to a copy of object do reflect in the original object. In python, this is implemented using “copy()” function.

Note: The copying process does not recurse and therefore won’t create copies of the nested child objects themselves.

What is Deep Copy ?


Deep copy is a process in which the copying process occurs recursively. It means first constructing a new collection object and then recursively populating it with copies of the child objects found in the original. In case of deep copy, a copy of object is copied in other object. It means that any changes made to a copy of object do not reflect in the original object. In python, this is implemented using “deepcopy()” function.

There are two problems often exist with deep copy operations that don’t exist with shallow copy operations:

    *   Recursive objects may cause a recursive loop.
    *   Because deep copy copies everything it may copy too much, such as data which is intended to be shared between copies.

How Deepcopy avoids such problems ?

    *   Keeping a “memo” dictionary of objects already copied during the current copying pass.
    *   Letting user-defined classes override the copying operation or the set of components copied.

Simple Example of Shallow copy without using copy module.

L = ["A", "B", "C", "D", [1, 2, 3], "E", "F"]

print("Original List Before Copy", L)
L1 = list(L)
print("Original List After Copy", L)

print("Copy List L1 Before Changes", L1)
L1[0] = "a"
L1[1] = "b"
L1[4][0] = 4
L1[4][1] = 5 
print("Copy List L1 After Changes", L1)
print("Original List L After Changes in L1 List", L)

Original List Before Copy ['A', 'B', 'C', 'D', [1, 2, 3], 'E', 'F']
Original List After Copy ['A', 'B', 'C', 'D', [1, 2, 3], 'E', 'F']
Copy List L1 Before Changes ['A', 'B', 'C', 'D', [1, 2, 3], 'E', 'F']
Copy List L1 After Changes ['a', 'b', 'C', 'D', [4, 5, 3], 'E', 'F']
Original List L After Changes in L1 List ['A', 'B', 'C', 'D', [4, 5, 3], 'E', 'F']

As you can see we made changes to L1 list on "0" and "1" position which are not reflected in original list but on "4" position we have another list object which was pointing to original object so changes made to this objects are reflected in original list L. So this is the disadvantage of shallow copy.


Above same example using copy module.


import copy
L = ["A", "B", "C", "D", [1, 2, 3], "E", "F"]

print("Original List Before Copy", L)
# using copy module
L1 = copy.copy(L)

print("Original List After Copy", L)
print("Copy List L1 Before Changes", L1)
L1[0] = "a"
L1[1] = "b"
L1[4][0] = 4
L1[4][1] = 5 
print("Copy List L1 After Changes", L1)
print("Original List L After Changes in L1 List", L)

Original List Before Copy ['A', 'B', 'C', 'D', [1, 2, 3], 'E', 'F']
Original List After Copy ['A', 'B', 'C', 'D', [1, 2, 3], 'E', 'F']
Copy List L1 Before Changes ['A', 'B', 'C', 'D', [1, 2, 3], 'E', 'F']
Copy List L1 After Changes ['a', 'b', 'C', 'D', [4, 5, 3], 'E', 'F']
Original List L After Changes in L1 List ['A', 'B', 'C', 'D', [4, 5, 3], 'E', 'F']

As we can see output remains same as above.

Above same example using Deepcopy.


import copy
L = ["A", "B", "C", "D", [1, 2, 3], "E", "F"]

print("Original List Before Copy", L)
# using deepcopy module
L1 = copy.deepcopy(L)

print("Original List After Copy", L)
print("Copy List L1 Before Changes", L1)
L1[0] = "a"
L1[1] = "b"
L1[4][0] = 4
L1[4][1] = 5 
print("Copy List L1 After Changes", L1)
print("Original List L After Changes in L1 List", L)

Original List Before Copy ['A', 'B', 'C', 'D', [1, 2, 3], 'E', 'F']
Original List After Copy ['A', 'B', 'C', 'D', [1, 2, 3], 'E', 'F']
Copy List L1 Before Changes ['A', 'B', 'C', 'D', [1, 2, 3], 'E', 'F']
Copy List L1 After Changes ['a', 'b', 'C', 'D', [4, 5, 3], 'E', 'F']
Original List L After Changes in L1 List ['A', 'B', 'C', 'D', [1, 2, 3], 'E', 'F']

As you can see in above result, using deepcopy changes made to L1 list are not reflected in original list not even in child objects at "4" position. This is the advantage of using deepcopy as it creates complete new copy of original object so changes made to original object or changes made to copied object does not affect each other.


Example of using copy and deepcopy Magic (Dunder) Methods.


import copy
class Employee(object):
    def __init__(self):
        self.name = "John"
        self.age = 35
    def __copy__(self):
        cls = self.__class__
        instance = cls.__new__(cls)
        instance.__dict__.update(self.__dict__)
        return instance
    def __deepcopy__(self, memo):
        cls = self.__class__
        instance = cls.__new__(cls)
        memo[id(self)] = instance
        for k, v in self.__dict__.items():
            setattr(instance, k, copy.deepcopy(v, memo))
        return instance

emp = Employee()
emp_1 = copy.copy(emp)

print(emp.age)
print(emp_1.age)

emp.age = 25
del emp.name
emp.salary = 25000
emp_2 = copy.deepcopy(emp)
emp_2.age
emp_2.name
emp_2.salary

35
35
25
AttributeError: 'Employee' object has no attribute 'name'
25000

Python Magic (Dunder) Methods

What are python magic methods ?

Magic methods (Dunder Methods) are special methods in python that you can define to add "magic" to your classes. Magic methods are always surrounded by double underscores (e.g. __init__ or __lt__). How to pronouce this methods like __init__ ?  "Double underscore init double underscore", but the ideal way is "dunder init dunder". That's why magic methods methods are sometimes called dunder methods.

what's magic about this methods? The answer is, you don't have to invoke it directly. The invocation is realized behind the scenes. When you create an instance of a class. A class can implement certain operations that are invoked by special syntax (such as arithmetic operations or subscripting and slicing) by defining methods with special names. This is Python’s approach to operator overloading, allowing classes to define their own behavior with respect to language operators.

Following are list of Magic (Dunder) Methods

Basic Customization

* object.__new__(cls[, ...])
* object.__init__(self[, ...])
* object.__del__(self)
* object.__repr__(self)
* object.__str__(self)
* object.__bytes__(self)
* object.__format__(self, formatstr)
* object.__hash__(self)
* object.__bool__(self)
* object.__init_subclass__(cls)
* object.__len__(self)
* object.__iter__(self)
* object.__reversed__(self)
* object.__contains__(self, item)
* object.__call__(self, *args, **kwargs)
* object.__enter__(self)
* object.__exit__(self, exc_type, exc_value, traceback)

Customizing Attribute Access

* object.__getattr__(self, name)
* object.__getattribute__(self, name)
* object.__setattr__(self, name, value)
* object.__delattr__(self, name)
* object.__dir__(self)

Implementing Descriptors

* object.__get__(self, instance, owner=None)
* object.__set__(self, instance, value)
* object.__delete__(self, instance)
* object.__set_name__(self, owner, name)
* object.__slots__

Binary Operators

* object.__add__(self, other)
* object.__sub__(self, other)
* object.__mul__(self, other)
* object.__floordiv__(self, other)
* object.__truediv__(self, other)
* object.__mod__(self, other)
* object.__pow__(self, other[, modulo])
* object.__lshift__(self, other)
* object.__rshift__(self, other)
* object.__and__(self, other)
* object.__xor__(self, other)
* object.__or__(self, other)

Extended Assignments

* object.__iadd__(self, other)
* object.__isub__(self, other)
* object.__imul__(self, other)
* object.__idiv__(self, other)
* object.__ifloordiv__(self, other)
* object.__imod__(self, other)
* object.__ipow__(self, other[, modulo])
* object.__ilshift__(self, other)
* object.__irshift__(self, other)
* object.__iand__(self, other)
* object.__ixor__(self, other)
* object.__ior__(self, other)

Unary Operators

* object.__neg__(self)
* object.__pos__(self)
* object.__abs__(self)
* object.__invert__(self)
* object.__complex__(self)
* object.__int__(self)
* object.__long__(self)
* object.__float__(self)
* object.__oct__(self)
* object.__hex__(self)
* object.__round__(self[, ndigits])
* object.__trunc__(self)
* object.__floor__(self)
* object.__ceil__(self)

Comparison Operators

* object.__lt__(self, other)
* object.__le__(self, other)
* object.__eq__(self, other)
* object.__ne__(self, other)
* object.__ge__(self, other)
* object.__gt__(self, other)

object.new(cls[, ...])

This method is called to create a new instance of class cls. __new__() is a static method that takes the class of which an instance was requested as its first argument. The remaining arguments are those passed to the object constructor expression (the call to the class). The return value of __new__() should be the new object instance (usually an instance of cls).

Typical implementations create a new instance of the class by invoking the superclass’s __new__() method using super(currentclass, cls).__new__(cls, [,….]) with appropriate arguments and then modifying the newly-created instance as necessary before returning it.

If __new__() is invoked during object construction and it returns an instance or subclass of cls, then the new instance’s __init__() method will be invoked like __init__(self[, ...]), where self is the new instance and the remaining arguments are the same as were passed to the object constructor.

If __new__() does not return an instance of cls, then the new instance’s __init__() method will not be invoked.

__new__() is intended mainly to allow subclasses of immutable types (like int, str, or tuple) to customize instance creation. It is also commonly overridden in custom metaclasses in order to customize class creation.
class Employee(object):
    def __new__(cls, *args, **kwargs):
        print("Creating Instance")
        name, age = args
        if age > 50:
            print("You can not work here")
            return None
        instance = super(Employee, cls).__new__(cls)
        return instance

    def __init__(self, name, age):
        self.name = name
        self.age = age

emp_1 = Employee("John", 25)
print(emp_1)
emp_2 = Employee("Jack", 51)
print(emp_2)
Creating Instance
<main.Employee object at 0x7fd216029310>
Creating Instance
You can not work here
None

Implementing Singleton design pattern using new method.

class Connection(object):
    _instance = None
    def __new__(cls, *args, **kwargs):
        if not cls._instance:
            cls._instance = object.__new__(cls)
        return cls._instance
    def __init__(self, host, port, user, password):
        self.host = host
        self.port = port
        self.user = user
        self.password = password

conn_1 = Connection("localhost", 5432, "user", "pass")
print(conn_1)
print(conn_1.user)
conn_2 = Connection("newhost", 5433, "user2", "pass2")
print(conn_2)
print(conn_2.user)
print(conn_1.user)
<main.Connection object at 0x7fd215919a00>
user
<main.Connection object at 0x7fd215919a00>
user2
user2


object.__init__(self[, ...])

This method is called after the instance has been created (by __new__()), but before it is returned to the caller. The arguments are those passed to the class constructor expression. If a base class has an __init__() method, the derived class’s __init__() method, if any, must explicitly call it to ensure proper initialization of the base class part of the instance; for example: super().__init__([args...]). Because __new__() and __init__() work together in constructing objects (__new__() to create it, and __init__() to customize it), no non-None value may be returned by __init__(); doing so will cause a TypeError to be raised at runtime.

When an object is created from a class and it allows the class to initialize the attributes of the class.
class Employee(object):
    def __init__(self, name, age):
        self.name = name
        self.age = age

emp_1 = Employee("John", 25)
print(emp_1.name)
print(emp_1.age)
emp_2 = Employee("Jack", 51)
print(emp_2.name)
print(emp_2.age)
John
25
Jack
51


object.__del__(self)

This method is called when the instance is about to be destroyed. This is also called a finalizer or a destructor. If a base class has a __del__() method, the derived class’s __del__() method, if any, must explicitly call it to ensure proper deletion of the base class part of the instance. It is possible for the __del__() method to postpone destruction of the instance by creating a new reference to it. This is called object resurrection.
class Employee(object):
    def __init__(self, name, age):
        print("Initializing Class")
        self.name = name
        self.age = age
    def __del__(self):
        print("Destroying Class")

emp_1 = Employee("John", 25)
print(emp_1)
del emp_1
print(emp_1)
<main.Employee object at 0x7fd2159316a0>
Destroying Class
NameError: name 'emp_1' is not defined


object.__repr__(self)

This method is called by the repr() built-in function to compute the “official” string representation of an object. If at all possible, this should look like a valid Python expression that could be used to recreate an object with the same value (given an appropriate environment). If this is not possible, a string of the form <...some useful description...> should be returned. The return value must be a string object. If a class defines __repr__() but not __str__(), then __repr__() is also used when an “informal” string representation of instances of that class is required. This is typically used for debugging, so it is important that the representation is information-rich and unambiguous.
class Employee(object):
    def __init__(self, name, age):
        self.name = name
        self.age = age
    def __repr__(self):
        return "<Employee object Employee({0}, {1})>".format(self.name, self.age)

emp_1 = Employee("John", 25)
print(repr(emp_1))
<Employee object Employee(John, 25)>


object.__str__(self)

This method is called by str(object) and the built-in functions format() and print() to compute the “informal” or nicely printable string representation of an object. The return value must be a string object. This method differs from object.__repr__() in that there is no expectation that __str__() return a valid Python expression: a more convenient or concise representation can be used. The default implementation defined by the built-in type object calls object.__repr__().
class Employee(object):
    def __init__(self, name, age):
        self.name = name
        self.age = age
    def __str__(self):
        return "Employee name={0} age={1}".format(self.name, self.age)

emp_1 = Employee("John", 25)
print(str(emp_1))
Employee name=John age=25


object.__bytes__(self)

This method is called by bytes to compute a byte-string representation of an object. This should return a bytes obj
ect.
class Employee(object):
    def __init__(self, name, age):
        self.name = name
        self.age = age
    def __bytes__(self):
        return b"Employee name=%b age=%b" % (bytes(self.name.encode()), bytes(self.age))

e = Employee("John", 25)
bytes(e)
b'Employee name=John age=\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'


object.__format__(self, format_spec)

This method is called by the format() built-in function, and by extension, evaluation of formatted string literals and the str.format() method, to produce a “formatted” string representation of an object. The return value must be a string object.
class Employee(object):
    def __init__(self, name, age):
        self.name = name
        self.age = age
    def __format__(self, format):
        if format == "age":
            return str(self.age)
        elif format == "name":
            return self.name
        else:
            return self.name + ", " + str(self.age)

e = Employee("John", 25)
format(e, "age")
format(e, "name")
format(e)
'25'
'John'
'John, 25'


object.__hash__(self)

This method is called by built-in function hash() and for operations on members of hashed collections including set, frozenset, and dict. __hash__() should return an integer. The only required property is that objects which compare equal have the same hash value; it is advised to mix together the hash values of the components of the object that also play a part in comparison of objects by packing them into a tuple and hashing the tuple.
class Employee(object):
    def __init__(self, name, age):
        self.name = name
        self.age = age
    def __hash__(self):
        return hash((self.name, self.age))

e = Employee("John", 25)
hash(e)
117175504876153979


object.__bool__(self)

This method is called to implement truth value testing and the built-in operation bool(). It should return False or True. When this method is not defined, __len__() is called, if it is defined, and the object is considered true if its result is nonzero. If a class defines neither __len__() nor __bool__(), all its instances are considered true.
class FalseObject(object):
    def __bool__(self):
        return False

fo = FalseObject()
bool(fo)

if fo:
    print("Not False Object")
else:
    print("Yes it is False Object")
False
Yes it is False Object


object.__init_subclass__(cls)

Whenever a class inherits from another class, __init_subclass__ is called on that class. This way, it is possible to write classes which change the behavior of subclasses. This is closely related to class decorators, but where class decorators only affect the specific class they’re applied to, __init_subclass__ solely applies to future subclasses of the class defining the method.
class Engine(object):
    def __init_subclass__(cls, *args, **kwargs):
        cls.hp = 12
        cls.cc = 1000
        super().__init_subclass__(*args, **kwargs)

class Car(Engine):
    def __init__(self, hp=None, cc=None):
        if hp:
            self.hp = hp
        if cc:
            self.cc = cc

class Truck(Engine):
    def __init__(self, hp=None, cc=None):
        if hp:
            self.hp = hp
        if cc:
            self.cc = cc

car = Car()
car.hp
car.cc

truck = Truck(15, 2000)
truck.hp
truck.cc
12
1000
15
2000


object.__len__(self)

Called to implement the built-in function len(). Should return the length of the object, an integer >= 0. Also, an object that doesn’t define a __bool__() method and whose __len__() method returns zero is considered to be false in a Boolean context.
class Employee(object):
    def __init__(self):
        self.name = "John Connor"
        self.age = 25
    def __len__(self):
        return len(self.name)

e = Employee()
print("Lenght of Employee name is", len(e))

class Employee(object):
    def __init__(self):
        self.name = "John Connor"
        self.age = 25
    def __len__(self):
        return 0

emp = Employee()

if emp:
    print("This is True")
else:
    print("This is False, as __len__() returns 0")
Lenght of Employee name is 11
This is False, as len() returns 0


object.__iter__(self)

This method is called when an iterator is required for a container. This method should return a new iterator object that can iterate over all the objects in the container. For mappings, it should iterate over the keys of the container.
class Employee(object):
    def __init__(self):
        self.name = "John"
        self.age = 25
        self.departments = ["Accounts", "HR", "Back Office"]
    def __iter__(self):
        return iter(self.departments)

emp = Employee()
for dept in emp:
    print("Employee belongs to {} department".format(dept))
Employee belongs to Accounts department
Employee belongs to HR department
Employee belongs to Back Office department


object.__reversed__(self)

This method is called by the reversed() built-in to implement reverse iteration. It should return a new iterator object that iterates over all the objects in the container in reverse order.
class Employee(object):
    def __init__(self):
        self.name = "John"
        self.age = 25
        self.departments = ["Accounts", "HR", "Back Office"]
    def __reversed__(self):
        return reversed(self.departments)

emp = Employee()
for dept in reversed(emp):
    print("Employee belongs to {} department".format(dept))
Employee belongs to Back Office department
Employee belongs to HR department
Employee belongs to Accounts department


object.__contains__(self, item)

This method is Called to implement membership test operators. Should return true if item is in self, false otherwise. For mapping objects, this should consider the keys of the mapping rather than the values or the key-item pairs.
class Employee(object):
    def __init__(self):
        self.name = "John"
        self.age = 25
        self.departments = ["Accounts", "HR", "Back Office"]
    def __contains__(self, dept):
        return dept in self.departments

emp = Employee()

print("Employee is part of HR department", "HR" in emp)
print("Employee is part of Accounts department", "Accounts" in emp)
print("Employee is part of Operations department", "Operations" in emp)
Employee is part of HR department True
Employee is part of Accounts department True
Employee is part of Operations department False


object.__call__(self, *args, **kwargs)

The __call__ method can be used to turn the instances of the class into callables. Functions are callable objects. A callable object is an object which can be used and behaves like a function but might not be a function. By using the __call__ method it is possible to define classes in a way that the instances will be callable objects. The __call__ method is called, if the instance is called "like a function", i.e. using brackets.
class CalculateInterest(object):
    def __init__(self):
        self.number_of_payments = 12
        self.rate_of_interest = 0.084
    def __call__(self, principal_amount):
        return (self.rate_of_interest / self.number_of_payments) * principal_amount

loan_amounts = [10000, 20000, 30000, 40000]
interest = CalculateInterest()
for principal_amount in loan_amounts:
    print("Borrower will pay ${0} as interest for principal loan amount ${1}".format(interest(principal_amount), principal_amount))
Borrower will pay $70.0 as interest for principal loan amount $10000
Borrower will pay $140.0 as interest for principal loan amount $20000
Borrower will pay $210.0 as interest for principal loan amount $30000
Borrower will pay $280.0 as interest for principal loan amount $40000


object.__enter__(self)

This method is used with "with statement" and return value to the target specified in the as clause of the statement.


object.__exit__(self, exc_type, exc_value, traceback)

This method is called when with statement is exited. The parameters describe the exception that caused the context to be exited. If the context was exited without an exception, all three arguments will be None.
class DBConnection(object):
    def __init__(self):
        self.host = "localhost"
        self.port = 5660
        self.is_connected = False
    def __enter__(self):
        print("creating database connection to {0} on port {1}".format(self.host, self.port))
        self.is_connected = True
        print(self.is_connected)
    def __exit__(self, exc_type=None, exc_value=None, traceback=None):
        print("closing database connection")
        self.is_connected = False
        print(self.is_connected)

db_conn = DBConnection()
with db_conn:
    print("Executing queries")
creating database connection to localhost on port 5660
True
Executing queries
closing database connection
False


object.__getattr__(self, name)

This method is called when the default attribute access fails with an AttributeError, because name is not an instance attribute or an attribute in the class tree. This method should either return the (computed) attribute value or raise an AttributeError exception.

Note that if the attribute is found through the normal mechanism, __getattr__() is not called. This is done both for efficiency reasons and because otherwise __getattr__() would have no way to access other attributes of the instance.

You can define behavior for when a user attempts to access an attribute that doesn't exist (either at all or yet). This can be useful for catching and redirecting common misspellings, giving warnings about using deprecated attributes (you can still choose to compute and return that attribute, if you wish), or deftly handing an AttributeError. It only gets called when a nonexistent attribute is accessed
class Shirt(object):
    def __init__(self):
        self.size = "XL"
        self.color = "Black"
        self.pattern = "Plain"
    def __getattr__(self, key):
        if key == "dimensions":
            return self.size
        elif key == "colour":
            return self.color
        elif key == "design":
            return self.pattern
        else:
            raise AttributeError

s = Shirt()
s.color
s.colour
s.size
s.dimensions
s.pattern
s.design
s.something
'Black'
'Black'
'XL'
'XL'
'Plain'
'Plain'
AttributeError


object.__getattribute__(self, name)

If your class defines a __getattribute__() method, Python will call it on every reference to any attribute or method name. The __getattribute__() method is called to provide a value for object.X; Even after explicitly setting object.X, the __getattribute__() method is still called to provide a value for object.X; If present, the __getattribute__() method is called unconditionally for every attribute and method lookup, even for attributes that you explicitly set after creating an instance. If your class defines a __getattribute__() method, you probably also want to define a __setattr__() method and coordinate between them to keep track of attribute values. Otherwise, any attributes you set after creating an instance will disappear into a black hole. You need to be extra careful with the __getattribute__() method, because it is also called when Python looks up a method name on your class. If class defines a __getattribute__() method which always raises an AttributeError exception. No attribute or method lookups will succeed. When you call object.X(), Python looks for a X() method in the class. This lookup goes through the __getattribute__() method, because all attribute and method lookups go through the __getattribute__() method.

If a class defines both "__getattr__" and "__getattribute__" method then python first calls "__getattribute__" method to get attribute and then it calls "__getattr__".
class Shirt(object):
    def __init__(self):
        self.size = "XL"
        self.color = "Black"
        self.pattern = "Plain"
    def __getatrribute__(self, key):
        if key == "size":
            return self.size
        elif key == "color":
            return self.color
        elif key == "pattern":
            return self.pattern
        else:
            raise AttributeError

shirt = Shirt()
shirt.size
shirt.color
shirt.pattern

'XL'
'Black'
'Plain'

class X(object):
    def __getattr__(self, key):
        print("called __getattr__")
        if key == "a":
            return "a"
        else:
            print("Raise error from __getattr__")
            raise AttributeError
    def __getattribute__(self, key):
        print("called __getattribute__")
        if key == "b":
            return "b"
        else:
            print("Raise error from __getattribute__")
            raise AttributeError

x = X()
x.a
x.b
x.c

called getattribute Raise error from getattribute
called getattr
'a'
called getattribute
'b'
called getattribute
Raise error from getattribute
called getattr
Raise error from getattr
AttributeError


object.__setattr__(self, name, value)

This method is called when an attribute assignment is attempted. This is called instead of the normal mechanism. name is the attribute name, value is the value to be assigned to it. It allows you to define behavior for assignment to an attribute regardless of whether or not that attribute exists, meaning you can define custom rules for any changes in the values of attributes.
class Employee(object):
    def __init__(self):
        self.name = "John"
        self.age = 25
    def __setattr__(self, key, value):
        if key == "age":
            if value > 50:
                print("You can not set age greater than 50")
            else:
                self.__dict__[key] = value
        else:
            self.__dict__[key] = value

e = Employee()
e.name
e.age
e.age = 51
e.age = 35
e.name = "Jack"
e.salary = 10000
e.name
e.age
e.salary

'John'
-25
You can not set age greater than 50
'Jack'
35
10000


object.__delattr__(self, name)

This method is used for attribute deletion instead of assignment. This should only be implemented if del obj.name is meaningful for the object.
class Employee(object):
    def __init__(self):
        self.name = "John"
        self.age = 25
        self.phone = 8888888888
    def __delattr__(self, key):
        if key == "age":
            print("You can not delete employees age")
        else:
            del self.__dict__[key]
            print("Deleted {} for employee".format(key))

e = Employee()
del e.name
del e.age
del e.phone
e.age
Deleted name for employee
You can not delete employees age
Deleted phone for employee
25


object.__dir__(self)

Defines behavior for when dir() is called on an instance of your class. This method should return a list of attributes for the user. Typically, implementing __dir__ is unnecessary, but it can be vitally important for interactive use of your classes.

class Employee(object):
    def __init__(self):
        self.name = "John"
        self.age = 25
        self.phone = 8888888888
    def __dir__(self):
        dirs = super(Employee, self).__dir__()
        dirs.append("custom_attr")
        self.custom_attr = "I am custom attribute"
        return dirs

e = Employee()
dir(e)
['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'age', 'custom_attr', 'name', 'phone']
'I am custom attribute'

Following are descriptor methods. Descriptors are classes which, when accessed through either getting, setting, or deleting, can also alter other objects. Descriptors aren't meant to stand alone; rather, they're meant to be held by an owner class. Descriptors can be useful when building object-oriented databases or classes that have attributes whose values are dependent on each other. Descriptors are particularly useful when representing attributes in several different units of measurement or representing computed attributes


object.__get__(self, instance, owner=None)

This method is called to get the attribute of the owner class or of an instance of that class. The optional owner argument is the owner class, while instance is the instance that the attribute was accessed through, or None when the attribute is accessed through the owner.
class SalaryPerMonth(object):    
    def __get__(self, instance, owner=None):
        salary = instance.salary/12
        return salary

class Employee(object):
    salary_per_month = SalaryPerMonth()
    def __init__(self):
        self.name = "John"
        self.age = 25
        self.phone = 8888888888
        self.salary = 100000

e = Employee()
e.salary
e.salary_per_month    


object.__set__(self, instance, value)

Define behavior for when the descriptor's value is changed. instance is the instance of the owner class and value is the value to set the descriptor to.
class Age(object):
    def __get__(self, instance, owner=None):
        return int(instance.age)
    def __set__(self, instance, age):
        instance.age = int(age)

class Employee(object):
    def __init__(self):
        self.name = "John"
        self.age = Age()

e = Employee()
e.age
e.age = 25
e.age
<main.Age object at 0x7fd5fb7327c0>
25


object.__delete__(self, instance)

Define behavior for when the descriptor's value is deleted. instance is the instance of the owner object.
class Age(object):
    def __get__(self, instance, owner=None):
        return int(instance.age)
    def __set__(self, instance, age):
        instance.age = int(age)
    def __delete__(self, instance):
        if instance.age:
            print("Deleting age")
            del instance.age
        else:
            raise AttributeError

class Employee(object):
    def __init__(self):
        self.name = "John"
        self.age = Age()

e = Employee()
e.age = 25
del e.age
del e.age
'Deleting age'
AttributeError: age


objects.__slots__(self)

Every class can have instance attributes. By default Python uses a dict to store an object’s instance attributes. This is really helpful as it allows setting arbitrary new attributes at runtime. A small classes with known attributes it might be a bottleneck. The dict wastes a lot of RAM. Python can’t just allocate a static amount of memory at object creation to store all the attributes. Therefore it uses a lot of RAM if you create a lot of objects. Still there is a way to circumvent this issue. It involves the usage of __slots__ to tell Python not to use a dict, and only allocate space for a fixed set of attributes. You can not assign new attributes this class.
class Employee(object):
    __slots__ = ['name', 'age']
    def __init__(self):
        self.name = "John"
        self.age = 25

e = Employee()
e.name
e.age
e.salary = 25000
'John'
25
AttributeError: 'Employee' object has no attribute 'salary'


object.__add__(self, other)

This magic method is used to overload "+" addition operator. When this method is defined it will implement a behaviour when two objects are added.
class Employee(object):
    def __init__(self, name, age, salary):
        self.name = name
        self.age = age
        self.salary = salary
    def __add__(self, other):
        return self.salary + other.salary

emp_1 = Employee("John", 35, 35000)
emp_2 = Employee("John", 30, 25000)

print ("Total salary of employees is ", emp_1 + emp_2)
Total salary of employees is 60000


object.__sub__(self, other)

This magic method is used to overload "-" substraction operator. When this method is defined it will implement a behaviour when two objects are substracted.
class Employee(object):
    def __init__(self, name, age, salary):
        self.name = name
        self.age = age
        self.salary = salary
    def __sub__(self, other):
        return self.salary - other.salary

emp_1 = Employee("John", 35, 35000)
emp_2 = Employee("John", 30, 25000)

print ("Difference in salary of employees is ", emp_1 - emp_2)
Difference in salary of employees is 10000


object.__mul__(self, other)

This magic method is used to overload "x" multiplication operator. When this method is defined it will implement a behaviour when two objects are multiplied.
class Employee(object):
    def __init__(self, name, age, salary):
        self.name = name
        self.age = age
        self.salary = salary
    def __mul__(self, months):
        return self.salary * months

emp_1 = Employee("John", 35, 35000)

print("Yearly package of employee is ", emp_1 * 12)
Yearly package of employee is 420000


object.__floordiv__(self, other)

This magic method is used to overload "//" floordiv operator. When this method is defined it will implement a behaviour when two objects are floordivided. Floor division returns the quotient(answer or result of division) in which the digits after the decimal point are removed. But if one of the operands(dividend and divisor) is negative, then the result is floored, i.e., rounded away from zero.
class Employee(object):
    def __init__(self, name, age, salary):
        self.name = name
        self.age = age
        self.salary = salary
    def __floordiv__(self, months):
        return self.salary // months

emp_1 = Employee("John", 35, 35000)

emp_1.salary / 12

print("Floor Division is ", emp_1 // 12)
2916.6666666666665
Floor Division is 2916


object.__truediv__(self, other)

The division operator (/) is implemented by these methods. The __truediv__() method is used when __future__.division is in effect, otherwise __div__() is used. If only one of these two methods is defined, the object will not support division in the alternate context; TypeError will be raised instead. This method returns a / b where 2/3 is .66 rather than 0. This is also known as “true” division.
class Employee(object):
    def __init__(self, name, age, salary):
        self.name = name
        self.age = age
        self.salary = salary
    def __truediv__(self, months):
        return self.salary / months

emp_1 = Employee("John", 35, 35000)

emp_1.salary / 12

print("True Division is ", emp_1 / 12)
2916.6666666666665
True Division is 2916.6666666666665


object.__mod__(self, other)

This method returns modulus of two objects.
class Employee(object):
    def __init__(self, name, age, salary):
        self.name = name
        self.age = age
        self.salary = salary
    def __mod__(self, months):
        return self.salary % months

emp_1 = Employee("John", 35, 35000)

print("Modulus is ", emp_1 % 12)
Modulus is 8


object.__pow__(self, other[, modulo])

This method returns a ** b, for a and b numbers.
class Employee(object):
    def __init__(self, name, age, salary):
        self.name = name
        self.age = age
        self.salary = salary
    def __pow__(self, months):
        return self.salary ** months

emp_1 = Employee("John", 35, 35000)

print("Power is ", emp_1 ** 2)
Power is 1225000000


object.__lshift__(self, other)

Bitwise left shift. This method shifts the bits of the first operand left by the specified number of bits.


object.__rshift__(self, other)

Bitwise right shift. This method shifts the bits of the first operand right by the specified number of bits.


object.__and__(self, other)

Returns the result of bitwise AND of two integers.


object.__or__(self, other)

Returns the result of bitwise OR of two integers.


object.__xor__(self, other)

Returns the result of bitwise XOR of two integers.
class Month(object):
    def __init__(self, months):
        self.months = months
    def __lshift__(self, num):
        return self.months << num
    def __rshift__(self, num):
        return self.months >> num
    def __and__(self, num):
        return self.months & num
    def __or__(self, num):
        return self.months | num
    def __xor__(self, num):
        return self.months ^ num

months = Month(12)
print("Left Shift is ", months << 1)
print("Right Shift is ", months >> 1)
print("Bitwise AND is ", months & 1)
print("Bitwise OR is ", months | 1)
print("Bitwise XOR is ", months ^ 1)
Left Shift is 24
Right Shift is 6
Left Shift is 24
Right Shift is 6
Bitwise AND is 0
Bitwise OR is 13
Bitwise XOR is 13
The following functions provide a more primitive access to in-place operators than the usual syntax does; for example, the statement x += y is equivalent to x = operator.iadd(x, y).


object.__iadd__(self, other)

iadd(a, b) is equivalent to a += b.


object.__isub__(self, other)

isub(a, b) is equivalent to a -= b.


object.__imul__(self, other)

imul(a, b) is equivalent to a *= b.


object.__idiv__(self, other)

idiv(a, b) is equivalent to a /= b.


object.__ifloordiv__(self, other)

ifloordiv(a, b) is equivalent to a //= b.


object.__imod__(self, other)

imod(a, b) is equivalent to a %= b.


object.__ipow__(self, other[, modulo])

ipow(a, b) is equivalent to a **= b.


object.__ilshift__(self, other)

ilshift(a, b) is equivalent to a <<= b.


object.__irshift__(self, other)

irshift(a, b) is equivalent to a >>= b.


object.__iand__(self, other)

iand(a, b) is equivalent to a &= b.


object.__ixor__(self, other)

ixor(a, b) is equivalent to a ^= b.


object.__ior__(self, other)

ior(a, b) is equivalent to a |= b.
class Month(object):
    def __init__(self, months):
        self.months = months
    def __iadd__(self, num):
        self.months += num
        return self.months
    def __isub__(self, num):
        self.months -= num
        return self.months
    def __imul__(self, num):
        self.months *= num
        return self.months
    def __ifloordiv__(self, num):
        self.months //= num
        return self.months
    def __imod__(self, num):
        self.months %= num
        return self.months
    def __ipow__(self, num):
        self.months **= num
        return self.months
    def __ilshift__(self, num):
        self.months <<= num
        return self.months
    def __irshift__(self, num):
        self.months >>= num
        return self.months
    def __iand__(self, num):
        self.months &= num
        return self.months
    def __ixor__(self, num):
        self.months ^= num
        return self.months
    def __ior__(self, num):
        self.months |= num
        return self.months

months = Month(12)
months += 2
print(months)

months = Month(12)
months -= 2
print(months)

months = Month(12)
months *= 2
print(months)

months = Month(12)
months //= 2
print(months)

months = Month(12)
months %= 1
print(months)

months = Month(12)
months **= 2
print(months)

months = Month(12)
months <<= 2
print(months)

months = Month(12)
months >>= 2
print(months)

months = Month(12)
months &= 2
print(months)

months = Month(12)
months ^= 2
print(months)

months = Month(12)
months |= 2
print(months)
14
10
24
6
0
144
48
3
0
14
14
Following methods are called to implement the unary arithmetic operations.


object.__neg__(self)

Returns obj negated (-obj).


object.__pos__(self)

Returns obj positive (+obj).


object.__abs__(self)

Return the absolute value of obj.


object.__invert__(self)

Return the bitwise inverse of the number obj. This is equivalent to ~obj.


object.__complex__(self)

This method is called to implement the built-in functions complex().


object.__int__(self)

This method is called to implement the built-in functions int().


object.__float__(self)

This method is called to implement the built-in functions float()


object.__oct__(self)

This method is called to implement the built-in functions oct(). Should return a string value.


object.__hex__(self)

This method is called to implement the built-in functions hex(). Should return a string value.
class Employee(object):
    def __init__(self):
        self.name = "John"
        self.age = 35
        self.salary = 35000
    def __index__(self):
        return self.age
    def __neg__(self):
        return -self.age
    def __pos__(self):
        return +self.age
    def __abs__(self):
        return abs(self.age)
    def __invert__(self):
        return ~self.age
    def __complex__(self):
        return complex(self.age)
    def __int__(self):
        return int(self.age)
    def __float__(self):
        return float(self.age)
    def __oct__(self):
        return oct(self.age)
    def __hex__(self):
        return hex(self.age)

e = Employee()
print("Negative representation of employee age is ", -e)
print("Positive representation of employee age is ", +e)
print("Absolute representation of employee age is ", abs(e))
print("Inverse representation of employee age is ", ~e)
print("Complex representation of employee age is ", complex(e))
print("Integer representation of employee age is ", int(e))
print("Float representation of employee age is ", float(e))
print("Octal representation of employee age is ", oct(e))
print("Hexadecimal representation of employee age is ", hex(e))
Negative representation of employee age is -35
Negative representation of employee age is 35
Absolute representation of employee age is 35
Inverse representation of employee age is -36
Complex representation of employee age is (35+0j)
Integer representation of employee age is 35
Float representation of employee age is 35.0
Octal representation of employee age is 0o43
Hexadecimal representation of employee age is 0x23

Folllowing methods are so-called “rich comparison” methods. The correspondence between operator symbols and method names is as follows: x<y calls x.lt(y), x<=y calls x.le(y), x==y calls x.eq(y), x!=y calls x.ne(y), x>y calls x.gt(y), and x>=y calls x.ge(y).
This method may return the singleton NotImplemented if it does not implement the operation for a given pair of arguments. By convention, False and True are returned for a successful comparison. However, these methods can return any value, so if the comparison operator is used in a Boolean context (e.g., in the condition of an if statement), Python will call bool() on the value to determine if the result is true or false.


object.__lt__(self, other)
Returns x < y


object.__le__(self, other)

Returns x <= y


object.__eq__(self, other)

Returns x == y


object.__ne__(self, other)

Returns x != y


object.__ge__(self, other)

Returns x >= y


object.__gt__(self, other)

Returns x > y
class Employee(object):
    def __init__(self):
        self.name = "John"
        self.age = 35
    def __lt__(self, num):
        return self.age < num
    def __le__(self, num):
        return self.age <= num
    def __eq__(self, num):
        return self.age == num
    def __ne__(self, num):
        return self.age != num
    def __ge__(self, num):
        return self.age >= num
    def __gt__(self, num):
        return self.age > num

e = Employee()

print("Employee age is less than 40", e < 40)
print("Employee age is less than equal to 30", e < 30)
print("Employee age is equal to 32", e == 32)
print("Employee age is not equal to 32", e != 32)
print("Employee age is greater than 32", e > 32)
print("Employee age is greater than equal to 36", e >= 36)
Employee age is less than 40 True
Employee age is less than equal to 30 False
Employee age is equal to 32 False
Employee age is not equal to 32 True
Employee age is greater than 32 True
Employee age is greater than equal to 36 False


It was all about python magic methods. If i missed any important magic method please add it to comment box.