Q:

When building datastructures by combining parts of old datastructures, does Python always do shallow copy?

If I create a new list out of some existing elements in a list, will python then shallow copy the items to the new list?
Here is a code example to clarify what I mean:
structure = [] 
structure.append([pd.read_excel("df1.csv"), item1]) #contains a large dataframe, df1, and some other object, item1
structure.append([pd.read_excel("df2.csv"), item2]) #analoguous

lst = [structure[0][0]["some_column1"], structure[1][0]["some_column2"]] #here we save just the dataframe

When doing other, similar operations of extracting stuff from existing datastructures and inserting it in new datastructures (without giving any explicit instruction to perform a deep copy), is there some kind of rule, when Python will shallow copy? Or can it happen in some cases, that the default copy mechanism is deep?

A:

Variables in Python are always references (smart pointers) to objects on the heap. There is no notion of objects living on the stack in Python like there is in C. Assigning a value a new name in Python does not copy it, it just adds a reference to the object.
A Python list is likewise a list of these references. Putting an existing heap object into a Python list just adds a reference to it. It doesn't copy it. The foo.copy() method or foo[:] full slice (where foo is a list) makes a shallow copy. Likewise list(foo) or [*foo] is a shallow copy. There is a deepcopy function in the copy module, if that's what you need.
The rules may be different for arrays. Pandas dataframes are based on Numpy arrays. While these can contain pointers (like for strings, which can be variable length), they can also contain the values in line without references, as is typical for arrays of numbers.
Multidimensional Numpy arrays of those types, unlike lists of lists, can't be shallow copied because they don't contain pointers at all. But, unlike Python's built in collections, slicing Numpy arrays typically makes a "view" to the same array, but there are certain cases where this is not possible and Numpy must make a copy.
Getting a value out of such an array means making a copy of the value as a heap object so that Python can have a direct reference to it. You can, of course, make a new reference to the array as a whole, by assigning it to a new variable or putting it in a list. But of course, this doesn't copy of the array at all.

So, just to confirm, that means that my lst is a deep copy of the parts of the pandas structures?

No (probably). You made a new list, but it (probably) contains references to the same columns objects as the dataframe, or at least "views" to the same.
Most operations in Pandas will make copies unless you use the inplace=True argument (which is not even available for some operations). Assignments to an index are in-place, of course, but beware of indirect chained indexing, which might end up assigning to a copy instead.
When getting by index, Pandas will try to use a view for efficiency, but like the Numpy arrays it's based on, this is not always possible (like when the object has multiple dtypes). I can't tell for sure which applies without knowing the structure of your dataframe, and maybe not even then, but my best guess it that would be a shallow copy. It can be hard to predict outside of simple cases, and may depend on the memory layout Pandas happened to pick, which is an implementation detail subject to change without notice. Don't count on it. Be explicit and use the copy methods.

