Instruction:
How do I use basic HTTP authentication with the Python Requests library?

I'm trying to use basic HTTP authentication in Python. I am using the Requests library: auth = requests.post('http://' + hostname, auth=HTTPBasicAuth(user, password)) request = requests.get('http://' + hostname + '/rest/applications') Response form auth variable: <<class 'requests.cookies.RequestsCookieJar'>[<Cookie JSESSIONID=cb10906c6219c07f887dff5312fb for appdynamics/controller>]> 200 CaseInsensitiveDict({'content-encoding': 'gzip', 'x-powered-by': 'JSP/2.2', 'transfer-encoding': 'chunked', 'set-cookie': 'JSESSIONID=cb10906c6219c07f887dff5312fb; Path=/controller; HttpOnly', 'expires': 'Wed, 05 Nov 2014 19:03:37 GMT', 'server': 'nginx/1.1.19', 'connection': 'keep-alive', 'pragma': 'no-cache', 'cache-control': 'max-age=78000', 'date': 'Tue, 04 Nov 2014 21:23:37 GMT', 'content-type': 'text/html;charset=ISO-8859-1'}) But when I try to get data from different location, I'm getting HTTP Status 401 error: <<class 'requests.cookies.RequestsCookieJar'>[]> 401 CaseInsensitiveDict({'content-length': '1073', 'x-powered-by': 'Servlet/3.0 JSP/2.2 (GlassFish Server Open Source Edition 3.1.2.2 Java/Oracle Corporation/1.7)', 'expires': 'Thu, 01 Jan 1970 00:00:00 UTC', 'server': 'nginx/1.1.19', 'connection': 'keep-alive', 'pragma': 'No-cache', 'cache-control': 'no-cache', 'date': 'Tue, 04 Nov 2014 21:23:37 GMT', 'content-type': 'text/html', 'www-authenticate': 'Basic realm="controller_realm"'}) As far as I understand, in the second request session parameters are not substituted.

Output: You need to use a session object and send the authentication each request. The session will also track cookies for you: session = requests.Session() session.auth = (user, password) auth = session.post('http://' + hostname) response = session.get('http://' + hostname + '/rest/applications')


Instruction:
Pythonic way to print list items

I would like to know if there is a better way to print all objects in a Python list than this : myList = [Person("Foo"), Person("Bar")] print("\n".join(map(str, myList))) Foo Bar I read this way is not really good : myList = [Person("Foo"), Person("Bar")] for p in myList: print(p) Isn't there something like : print(p) for p in myList If not, my question is... why ? If we can do this kind of stuff with comprehensive lists, why not as a simple statement outside a list ?

Output: Assuming you are using Python 3: print(*myList, sep='\n') This is a kind of unpacking. Details in the Python tutorial: Unpacking Argument Lists You can get the same behavior on Python 2 using from __future__ import print_function. With the print statement on Python 2 you will need iteration of some kind. Regarding your question about print(p) for p in myList not working, you can just use the following which does the same thing and is still simple: for p in myList: print p For a solution that uses '\n'.join(), I prefer list comprehensions and generators over map() so I would probably use the following: print '\n'.join(str(p) for p in myList)


Instruction:
Why does Pylint object to single-character variable names?

I'm still getting used to Python conventions and using Pylint to make my code more Pythonic, but I'm puzzled by the fact that Pylint doesn't like single character variable names. I have a few loops like this: for x in x_values: my_list.append(x) and when I run pylint, I'm getting Invalid name "x" for type variable (should match [a-z_][a-z0-9_]{2,30} -- that suggests that a valid variable name must be between 3 and 31 characters long, but I've looked through the PEP8 naming conventions and I don't see anything explicit regarding single lower case letters, and I do see a lot of examples that use them. Is there something I'm missing in PEP8 or is this a standard that is unique to Pylint?

Output: Pylint checks not only PEP8 recommendations. It has also its own recommendations, one of which is that a variable name should be descriptive and not too short. You can use this to avoid such short names: my_list.extend(x_values) Or tweak Pylint's configuration to tell Pylint what variable name are good.


Instruction:
How to write a multidimensional array to a text file?

In another question, other users offered some help if I could supply the array I was having trouble with. However, I even fail at a basic I/O task, such as writing an array to a file. Can anyone explain what kind of loop I would need to write a 4x11x14 numpy array to file? This array consist of four 11 x 14 arrays, so I should format it with a nice newline, to make the reading of the file easier on others. Edit: So I've tried the numpy.savetxt function. Strangely, it gives the following error: TypeError: float argument required, not numpy.ndarray I assume that this is because the function doesn't work with multidimensional arrays? Any solutions as I would like them within one file?

Output: If you want to write it to disk so that it will be easy to read back in as a numpy array, look into numpy.save. Pickling it will work fine, as well, but it's less efficient for large arrays (which yours isn't, so either is perfectly fine). If you want it to be human readable, look into numpy.savetxt. Edit: So, it seems like savetxt isn't quite as great an option for arrays with >2 dimensions... But just to draw everything out to it's full conclusion: I just realized that numpy.savetxt chokes on ndarrays with more than 2 dimensions... This is probably by design, as there's no inherently defined way to indicate additional dimensions in a text file. E.g. This (a 2D array) works fine import numpy as np x = np.arange(20).reshape((4,5)) np.savetxt('test.txt', x) While the same thing would fail (with a rather uninformative error: TypeError: float argument required, not numpy.ndarray) for a 3D array: import numpy as np x = np.arange(200).reshape((4,5,10)) np.savetxt('test.txt', x) One workaround is just to break the 3D (or greater) array into 2D slices. E.g. x = np.arange(200).reshape((4,5,10)) with open('test.txt', 'w') as outfile: for slice_2d in x: np.savetxt(outfile, slice_2d) However, our goal is to be clearly human readable, while still being easily read back in with numpy.loadtxt. Therefore, we can be a bit more verbose, and differentiate the slices using commented out lines. By default, numpy.loadtxt will ignore any lines that start with # (or whichever character is specified by the comments kwarg). (This looks more verbose than it actually is...) import numpy as np # Generate some test data data = np.arange(200).reshape((4,5,10)) # Write the array to disk with open('test.txt', 'w') as outfile: # I'm writing a header here just for the sake of readability # Any line starting with "#" will be ignored by numpy.loadtxt outfile.write('# Array shape: {0}\n'.format(data.shape)) # Iterating through a ndimensional array produces slices along # the last axis. This is equivalent to data[i,:,:] in this case for data_slice in data: # The formatting string indicates that I'm writing out # the values in left-justified columns 7 characters in width # with 2 decimal places. np.savetxt(outfile, data_slice, fmt='%-7.2f') # Writing out a break to indicate different slices... outfile.write('# New slice\n') This yields: # Array shape: (4, 5, 10) 0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00 11.00 12.00 13.00 14.00 15.00 16.00 17.00 18.00 19.00 20.00 21.00 22.00 23.00 24.00 25.00 26.00 27.00 28.00 29.00 30.00 31.00 32.00 33.00 34.00 35.00 36.00 37.00 38.00 39.00 40.00 41.00 42.00 43.00 44.00 45.00 46.00 47.00 48.00 49.00 # New slice 50.00 51.00 52.00 53.00 54.00 55.00 56.00 57.00 58.00 59.00 60.00 61.00 62.00 63.00 64.00 65.00 66.00 67.00 68.00 69.00 70.00 71.00 72.00 73.00 74.00 75.00 76.00 77.00 78.00 79.00 80.00 81.00 82.00 83.00 84.00 85.00 86.00 87.00 88.00 89.00 90.00 91.00 92.00 93.00 94.00 95.00 96.00 97.00 98.00 99.00 # New slice 100.00 101.00 102.00 103.00 104.00 105.00 106.00 107.00 108.00 109.00 110.00 111.00 112.00 113.00 114.00 115.00 116.00 117.00 118.00 119.00 120.00 121.00 122.00 123.00 124.00 125.00 126.00 127.00 128.00 129.00 130.00 131.00 132.00 133.00 134.00 135.00 136.00 137.00 138.00 139.00 140.00 141.00 142.00 143.00 144.00 145.00 146.00 147.00 148.00 149.00 # New slice 150.00 151.00 152.00 153.00 154.00 155.00 156.00 157.00 158.00 159.00 160.00 161.00 162.00 163.00 164.00 165.00 166.00 167.00 168.00 169.00 170.00 171.00 172.00 173.00 174.00 175.00 176.00 177.00 178.00 179.00 180.00 181.00 182.00 183.00 184.00 185.00 186.00 187.00 188.00 189.00 190.00 191.00 192.00 193.00 194.00 195.00 196.00 197.00 198.00 199.00 # New slice Reading it back in is very easy, as long as we know the shape of the original array. We can just do numpy.loadtxt('test.txt').reshape((4,5,10)). As an example (You can do this in one line, I'm just being verbose to clarify things): # Read the array from disk new_data = np.loadtxt('test.txt') # Note that this returned a 2D array! print new_data.shape # However, going back to 3D is easy if we know the # original shape of the array new_data = new_data.reshape((4,5,10)) # Just to check that they're the same... assert np.all(new_data == data)


Instruction:
Import pandas dataframe column as string not int

I would like to import the following csv as strings not as int64. Pandas read_csv automatically converts it to int64, but I need this column as string. ID 00013007854817840016671868 00013007854817840016749251 00013007854817840016754630 00013007854817840016781876 00013007854817840017028824 00013007854817840017963235 00013007854817840018860166 df = read_csv('sample.csv') df.ID >> 0 -9223372036854775808 1 -9223372036854775808 2 -9223372036854775808 3 -9223372036854775808 4 -9223372036854775808 5 -9223372036854775808 6 -9223372036854775808 Name: ID Unfortunately using converters gives the same result. df = read_csv('sample.csv', converters={'ID': str}) df.ID >> 0 -9223372036854775808 1 -9223372036854775808 2 -9223372036854775808 3 -9223372036854775808 4 -9223372036854775808 5 -9223372036854775808 6 -9223372036854775808 Name: ID

Output: Just want to reiterate this will work in pandas >= 0.9.1: In [2]: read_csv('sample.csv', dtype={'ID': object}) Out[2]: ID 0 00013007854817840016671868 1 00013007854817840016749251 2 00013007854817840016754630 3 00013007854817840016781876 4 00013007854817840017028824 5 00013007854817840017963235 6 00013007854817840018860166 I'm creating an issue about detecting integer overflows also. EDIT: See resolution here: https://github.com/pydata/pandas/issues/2247 Update as it helps others: To have all columns as str, one can do this (from the comment): pd.read_csv('sample.csv', dtype = str) To have most or selective columns as str, one can do this: # lst of column names which needs to be string lst_str_cols = ['prefix', 'serial'] # use dictionary comprehension to make dict of dtypes dict_dtypes = {x : 'str' for x in lst_str_cols} # use dict on dtypes pd.read_csv('sample.csv', dtype=dict_dtypes)


Instruction:
Is there an easy way to pickle a python function (or otherwise serialize its code)?

I'm trying to transfer a function across a network connection (using asyncore). Is there an easy way to serialize a python function (one that, in this case at least, will have no side effects) for transfer like this? I would ideally like to have a pair of functions similar to these: def transmit(func): obj = pickle.dumps(func) [send obj across the network] def receive(): [receive obj from the network] func = pickle.loads(s) func()

Output: You could serialise the function bytecode and then reconstruct it on the caller. The marshal module can be used to serialise code objects, which can then be reassembled into a function. ie: import marshal def foo(x): return x*x code_string = marshal.dumps(foo.__code__) Then in the remote process (after transferring code_string): import marshal, types code = marshal.loads(code_string) func = types.FunctionType(code, globals(), "some_func_name") func(10) # gives 100 A few caveats: marshal's format (any python bytecode for that matter) may not be compatable between major python versions. Will only work for cpython implementation. If the function references globals (including imported modules, other functions etc) that you need to pick up, you'll need to serialise these too, or recreate them on the remote side. My example just gives it the remote process's global namespace. You'll probably need to do a bit more to support more complex cases, like closures or generator functions.


Instruction:
Dynamically updating plot in matplotlib

I am making an application in Python which collects data from a serial port and plots a graph of the collected data against arrival time. The time of arrival for the data is uncertain. I want the plot to be updated when data is received. I searched on how to do this and found two methods: Clear the plot and re-draw the plot with all the points again. Animate the plot by changing it after a particular interval. I do not prefer the first one as the program runs and collects data for a long time (a day for example), and redrawing the plot will be pretty slow. The second one is also not preferable as time of arrival of data is uncertain and I want the plot to update only when the data is received. Is there a way in which I can update the plot just by adding more points to it only when the data is received?

Output: Is there a way in which I can update the plot just by adding more point[s] to it... There are a number of ways of animating data in matplotlib, depending on the version you have. Have you seen the animation examples in the matplotlib documentation? The animation API defines a function FuncAnimation which animates a function in time. This function could just be the function you use to acquire your data. Each method basically sets the data property of the object being drawn, so doesn't require clearing the screen or figure. The data property can simply be extended, so you can keep the previous points and just keep adding to your line (or image or whatever you are drawing). Given that you say that your data arrival time is uncertain your best bet is probably just to do something like: import matplotlib.pyplot as plt import numpy hl, = plt.plot([], []) def update_line(hl, new_data): hl.set_xdata(numpy.append(hl.get_xdata(), new_data)) hl.set_ydata(numpy.append(hl.get_ydata(), new_data)) plt.draw() Then when you receive data from the serial port just call update_line.


Instruction:
Keep selected column as DataFrame instead of Series

When selecting a single column from a pandas DataFrame(say df.iloc[:, 0], df['A'], or df.A, etc), the resulting vector is automatically converted to a Series instead of a single-column DataFrame. However, I am writing some functions that takes a DataFrame as an input argument. Therefore, I prefer to deal with single-column DataFrame instead of Series so that the function can assume say df.columns is accessible. Right now I have to explicitly convert the Series into a DataFrame by using something like pd.DataFrame(df.iloc[:, 0]). This doesn't seem like the most clean method. Is there a more elegant way to index from a DataFrame directly so that the result is a single-column DataFrame instead of Series?

Output: As @Jeff mentions there are a few ways to do this, but I recommend using loc/iloc to be more explicit (and raise errors early if you're trying something ambiguous): In [10]: df = pd.DataFrame([[1, 2], [3, 4]], columns=['A', 'B']) In [11]: df Out[11]: A B 0 1 2 1 3 4 In [12]: df[['A']] In [13]: df[[0]] In [14]: df.loc[:, ['A']] In [15]: df.iloc[:, [0]] Out[12-15]: # they all return the same thing: A 0 1 1 3 The latter two choices remove ambiguity in the case of integer column names (precisely why loc/iloc were created). For example: In [16]: df = pd.DataFrame([[1, 2], [3, 4]], columns=['A', 0]) In [17]: df Out[17]: A 0 0 1 2 1 3 4 In [18]: df[[0]] # ambiguous Out[18]: A 0 1 1 3


Instruction:
How to install pip for Python 3 on Mac OS X?

OS X (Mavericks) has Python 2.7 stock installed. But I do all my own personal Python stuff with 3.3. I just flushed my 3.3.2 install and installed the new 3.3.3. So I need to install pyserial again. I can do it the way I've done it before, which is: Download pyserial from pypi untar pyserial.tgz cd pyserial python3 setup.py install But I'd like to do like the cool kids do, and just do something like pip3 install pyserial. But it's not clear how I get to that point. And just that point. Not interested (unless I have to be) in virtualenv yet.

Output: UPDATE: This is no longer necessary as of Python3.4. pip3 is installed as part of the general Python3 installation. I ended up posting this same question on the python mailing list, and got the following answer: # download and install setuptools curl -O https://bootstrap.pypa.io/ez_setup.py python3 ez_setup.py # download and install pip curl -O https://bootstrap.pypa.io/get-pip.py python3 get-pip.py Which solved my question perfectly. After adding the following for my own: cd /usr/local/bin ln -s ../../../Library/Frameworks/Python.framework/Versions/3.3/bin/pip pip So that I could run pip directly, I was able to: # use pip to install pip install pyserial or: # Don't want it? pip uninstall pyserial


Instruction:
What is the equivalent of "none" in django templates?

I want to see if a field/variable is none within a Django template. What is the correct syntax for that? This is what I currently have: {% if profile.user.first_name is null %} <p> -- </p> {% elif %} {{ profile.user.first_name }} {{ profile.user.last_name }} {% endif%} In the example above, what would I use to replace "null"?

Output: None, False and True all are available within template tags and filters. None, False, the empty string ('', "", """""") and empty lists/tuples all evaluate to False when evaluated by if, so you can easily do {% if profile.user.first_name == None %} {% if not profile.user.first_name %} A hint: @fabiocerqueira is right, leave logic to models, limit templates to be the only presentation layer and calculate stuff like that in you model. An example: # someapp/models.py class UserProfile(models.Model): user = models.OneToOneField('auth.User') # other fields def get_full_name(self): if not self.user.first_name: return return ' '.join([self.user.first_name, self.user.last_name]) # template {{ user.get_profile.get_full_name }}


Instruction:
py.test: error: unrecognized arguments: --cov=ner_brands --cov-report=term-missing --cov-config

when I am trying to run my test through command line py.test file_name.py I got this error: py.test: error: unrecognized arguments: --cov=ner_brands --cov-report=term-missing --cov-config How can I fix this?

Output: pytest-cov package is required if you want to pass --cov arguments to pytest. By default it is not passed though. Are you using a modified version of py.test? pip install pytest-cov would fix your issue.


Instruction:
Reading a file using a relative path in a Python project

Say I have a Python project that is structured as follows: project /data test.csv /package __init__.py module.py main.py __init__.py: from .module import test module.py: import csv with open("..data/test.csv") as f: test = [line for line in csv.reader(f)] main.py: import package print(package.test) When I run main.py I get the following error: C:\Users\Patrick\Desktop\project>python main.py Traceback (most recent call last): File "main.py", line 1, in <module> import package File "C:\Users\Patrick\Desktop\project\package\__init__.py", line 1, in <module> from .module import test File "C:\Users\Patrick\Desktop\project\package\module.py", line 3, in <module> with open("../data/test.csv") as f: FileNotFoundError: [Errno 2] No such file or directory: '../data/test.csv' However, if I run module.py from the package directory, I don’t get any errors. So it seems that the relative path used in open(...) is only relative to where the originating file is being run from (i.e __name__ == "__main__")? How can deal with this, using relative paths only?

Output: Relative paths are relative to current working directory. If you do not want your path to be relative, it must be absolute. But there is an often used trick to build an absolute path from current script: use its __file__ special attribute: from pathlib import Path path = Path(__file__).parent / "../data/test.csv" with path.open() as f: test = list(csv.reader(f)) This requires python 3.4+ (for the pathlib module). If you still need to support older versions, you can get the same result with: import csv import os.path my_path = os.path.abspath(os.path.dirname(__file__)) path = os.path.join(my_path, "../data/test.csv") with open(path) as f: test = list(csv.reader(f)) [2020 edit: python3.4+ should now be the norm, so I moved the pathlib version inspired by jpyams' comment first]


Instruction:
Implementing use of 'with object() as f' in custom class in python

I have to open a file-like object in python (it's a serial connection through /dev/) and then close it. This is done several times in several methods of my class. How I WAS doing it was opening the file in the constructor, and then closing it in the destructor. I'm getting weird errors though and I think it has to do with the garbage collector and such, I'm still not used to not knowing exactly when my objects are being deleted =\ The reason I was doing this is because I have to use tcsetattr with a bunch of parameters each time I open it and it gets annoying doing all that all over the place. So I want to implement an inner class to handle all that so I can use it doing with Meter('/dev/ttyS2') as m: I was looking online and I couldn't find a really good answer on how the with syntax is implemented. I saw that it uses the __enter__(self) and __exit(self)__ methods. But is all I have to do implement those methods and I can use the with syntax? Or is there more to it? Is there either an example on how to do this or some documentation on how it's implemented on file objects already that I can look at?

Output: Those methods are pretty much all you need for making the object work with with statement. In __enter__ you have to return the file object after opening it and setting it up. In __exit__ you have to close the file object. The code for writing to it will be in the with statement body. MODE = 'rb' class Meter(): def __init__(self, dev): self.dev = dev def __enter__(self): #ttysetattr etc goes here before opening and returning the file object self.fd = open(self.dev, MODE) return self def __exit__(self, exception_type, exception_value, exception_traceback): #Exception handling here self.fd.close() meter = Meter('/dev/tty0') with meter as m: #here you work with the file object. m.fd.read()


Instruction:
Why can't I use a list as a dict key in python? Exactly what can and cannot be used, and why?

I found that the following are all valid: >>> d = {} >>> d[None] = 'foo' >>> d[(1, 3)] = 'baz' Even a module can be used as a dict key: >>> import sys >>> d[sys] = 'bar' However, a list cannot, and neither can a tuple that contains a list: >>> d[[2]] = 'spam' Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: unhashable type: 'list' >>> d[(1, [3])] = 'qux' Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: unhashable type: 'list' Why does storing a list inside the tuple mean it can't be a dict key any more? After all, I could just as easily "hide" a list inside a module (and indeed, e.g. sys.path is a list already). I had some vague idea that that the key has to be "hashable" but I don't have a detailed understanding of what this means, or why there is such a limitation. What would go wrong if Python allowed using lists as keys, say, using their memory location as the hash?

Output: There's a good article on the topic in the Python wiki: Why Lists Can't Be Dictionary Keys. As explained there: What would go wrong if Python allowed using lists as keys, say, using their memory location as the hash? It would cause some unexpected behavior. Lists are generally treated as if their value was derived from their content's values, for instance when checking (in-)equality. Many would - understandably - expect that you can use any list [1, 2] to get the same key, where you'd have to keep around exactly the same list object. But lookup by value breaks as soon as a list used as a key is modified, and lookup by identity requires keeping track of that exact list object - which isn't an ordinary requirement for working with lists. Other objects, such as modules and object, make a much bigger deal out of their object identity anyway (when was the last time you had two distinct module objects called sys?), and are compared by that anyway. Therefore, it's less surprising - or even expected - that they, when used as dict keys, compare by identity in that case as well.


Instruction:
When to use Serializer's create() and ModelViewset's perform_create()

I want to clarify the given documentation of Django-rest-framework regarding the creation of a model object. So far I have found that there are 3 approaches on how to handle such events. The Serializer's create() method. Here is the documentation class CommentSerializer(serializers.Serializer): def create(self, validated_data): return Comment.objects.create(**validated_data) The ModelViewset create() method. Documentation class AccountViewSet(viewsets.ModelViewSet): queryset = Account.objects.all() serializer_class = AccountSerializer permission_classes = [IsAccountAdminOrReadOnly] The ModelViewset perform_create() method. Documentation class SnippetViewSet(viewsets.ModelViewSet): def perform_create(self, serializer): serializer.save(owner=self.request.user) These three approaches are important depending on your application environment. But when do we need to use each create() / perform_create() function? On the other hand, I found some accounts that two create methods were called for a single post request the ModelViewSet's create() and serializer's create().

Output: You would use create(self, validated_data) to add any extra details into the object before saving AND "prod" values into each model field just like **validated_data does. Ideally speaking, you want to do this form of "prodding" only in ONE location so the create method in your CommentSerializer is the best place. On top of this, you might want to also call external apis to create user accounts on their side just before saving your accounts into your own database. You should use this create function in conjunction withModelViewSet. Always think - "Thin views, Thick serializers". Example: def create(self, validated_data): email = validated_data.get("email", None) validated.pop("email") # Now you have a clean valid email string # You might want to call an external API or modify another table # (eg. keep track of number of accounts registered.) or even # make changes to the email format. # Once you are done, create the instance with the validated data return models.YourModel.objects.create(email=email, **validated_data) The create(self, request, *args, **kwargs) function in the ModelViewSet is defined in the CreateModelMixin class which is the parent of ModelViewSet. CreateModelMixin's main functions are these: from rest_framework import status from rest_framework.response import Response def create(self, request, *args, **kwargs): serializer = self.get_serializer(data=request.data) serializer.is_valid(raise_exception=True) self.perform_create(serializer) headers = self.get_success_headers(serializer.data) return Response(serializer.data, status=status.HTTP_201_CREATED, headers=headers) def perform_create(self, serializer): serializer.save() As you can see, the above create function takes care of calling validation on your serializer and producing the correct response. The beauty behind this, is that you can now isolate your application logic and NOT concern yourself about the mundane and repetitive validation calls and handling response output :). This works quite well in conjuction with the create(self, validated_data) found in the serializer (where your specific application logic might reside). Now you might ask, why do we have a separate perform_create(self, serializer) function with just one line of code!?!? Well, the main reason behind this is to allow customizeability when calling the save function. You might want to supply extra data before calling save (like serializer.save(owner=self.request.user) and if we didn't have perform_create(self, serializer), you would have to override the create(self, request, *args, **kwargs) and that just defeats the purpose of having mixins doing the heavy and boring work.


Instruction:
typing.Any vs object?

Is there any difference between using typing.Any as opposed to object in typing? For example: def get_item(L: list, i: int) -> typing.Any: return L[i] Compared to: def get_item(L: list, i: int) -> object: return L[i]

Output: Yes, there is a difference. Although in Python 3, all objects are instances of object, including object itself, only Any documents that the return value should be disregarded by the typechecker. The Any type docstring states that object is a subclass of Any and vice-versa: >>> import typing >>> print(typing.Any.__doc__) Special type indicating an unconstrained type. - Any object is an instance of Any. - Any class is a subclass of Any. - As a special case, Any and object are subclasses of each other. However, a proper typechecker (one that goes beyond isinstance() checks, and which inspects how the object is actually used in the function) can readily object to object where Any is always accepted. From the Any type documentation: Notice that no typechecking is performed when assigning a value of type Any to a more precise type. and Contrast the behavior of Any with the behavior of object. Similar to Any, every type is a subtype of object. However, unlike Any, the reverse is not true: object is not a subtype of every other type. That means when the type of a value is object, a type checker will reject almost all operations on it, and assigning it to a variable (or using it as a return value) of a more specialized type is a type error. and from the mypy documentation section Any vs. object: The type object is another type that can have an instance of arbitrary type as a value. Unlike Any, object is an ordinary static type (it is similar to Object in Java), and only operations valid for all types are accepted for object values. object can be cast to a more specific type, while Any really means anything goes and a type checker disengages from any use of the object (even if you later assign such an object to a name that is typechecked). You already painted your function into a an un-typed corner by accepting list, which comes down to being the same thing as List[Any]. The typechecker disengaged there and the return value no longer matters, but since your function accepts a list containing Any objects, the proper return value would be Any here. To properly participate in type-checked code, you need to mark your input as List[T] (a genericly typed container) for a typechecker to then be able to care about the return value. Which in your case would be T since you are retrieving a value from the list. Create T from a TypeVar: from typing import TypeVar, List T = TypeVar('T') def get_item(L: List[T], i: int) -> T: return L[i] or, using Python 3.12 or newer: def get_item[T](L: list[T], i: int) -> T: return L[i]


Instruction:
Is there a difference between "raise exception()" and "raise exception" without parenthesis?

Defining a parameterless exception: class MyException(Exception): pass When raised, is there any difference between: raise MyException and raise MyException() I couldn't find any; is it simply an overloaded syntax?

Output: The short answer is that both raise MyException and raise MyException() do the same thing. This first form auto instantiates your exception. The relevant section from the docs says: raise evaluates the first expression as the exception object. It must be either a subclass or an instance of BaseException. If it is a class, the exception instance will be obtained when needed by instantiating the class with no arguments. That said, even though the semantics are the same, the first form is microscopically faster, and the second form is more flexible (because you can pass it arguments if needed). The usual style that most people use in Python (i.e. in the standard library, in popular applications, and in many books) is to use raise MyException when there are no arguments. People only instantiate the exception directly when there some arguments need to be passed. For example: raise KeyError(badkey).


Instruction:
Random hash in Python

What is the easiest way to generate a random hash (MD5) in Python?

Output: A md5-hash is just a 128-bit value, so if you want a random one: import random hash = random.getrandbits(128) print("hash value: %032x" % hash) I don't really see the point, though. Maybe you should elaborate why you need this...


Instruction:
Connecting to Microsoft SQL server using Python

I am trying to connect to SQL through python to run some queries on some SQL databases on Microsoft SQL server. From my research online and on this forum the most promising library seems to be pyodbc. So I have made the following code import pyodbc conn = pyodbc.connect(init_string="driver={SQLOLEDB}; server=+ServerName+; database=+MSQLDatabase+; trusted_connection=true") cursor = conn.cursor() and get the following error Traceback (most recent call last): File "C:\Users...\scrap.py", line 3, in <module> conn = pyodbc.connect(init_string="driver={SQLOLEDB}; server=+ServerName+; database=+MSQLDatabase+; trusted_connection=true") pyodbc.Error: ('IM002', '[IM002] [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified (0) (SQLDriverConnect)') I have looked at the folowing posts and tried changing my driver to {sql server} and have connected using ODBC links before in SAS, which is partially what my above code is based on, so don't think I need to install anything else. pyodbc.Error: ('IM002', '[IM002] [unixODBC][Driver Manager]Data source name not found, and no default driver specified (0) (SQLDriverConnect)') Pyodbc - "Data source name not found, and no default driver specified" Thanks

Output: This is how I do it... import pyodbc cnxn = pyodbc.connect("Driver={SQL Server Native Client 11.0};" "Server=server_name;" "Database=db_name;" "Trusted_Connection=yes;") cursor = cnxn.cursor() cursor.execute('SELECT * FROM Table') for row in cursor: print('row = %r' % (row,)) Relevant resources: Connecting to SQL Server from Windows | pyodbc GitHub Python and Data : SQL Server as a data source for Python applications


Instruction:
Caching class attributes in Python

I'm writing a class in python and I have an attribute that will take a relatively long time to compute, so I only want to do it once. Also, it will not be needed by every instance of the class, so I don't want to do it by default in __init__. I'm new to Python, but not to programming. I can come up with a way to do this pretty easily, but I've found over and over again that the 'Pythonic' way of doing something is often much simpler than what I come up with using my experience in other languages. Is there a 'right' way to do this in Python?

Output: 3.8 ≤ Python @property and @functools.lru_cache have been combined into @cached_property. import functools class MyClass: @functools.cached_property def foo(self): print("long calculation here") return 21 * 2 3.2 ≤ Python < 3.8 You should use both @property and @functools.lru_cache decorators: import functools class MyClass: @property @functools.lru_cache() def foo(self): print("long calculation here") return 21 * 2 This answer has more detailed examples and also mentions a backport for previous Python versions. Python < 3.2 The Python wiki has a cached property decorator (MIT licensed) that can be used like this: import random # the class containing the property must be a new-style class class MyClass(object): # create property whose value is cached for ten minutes @cached_property(ttl=600) def randint(self): # will only be evaluated every 10 min. at maximum. return random.randint(0, 100) Or any implementation mentioned in the others answers that fits your needs. Or the above mentioned backport.


Instruction:
How to compute precision, recall, accuracy and f1-score for the multiclass case with scikit learn?

I'm working in a sentiment analysis problem the data looks like this: label instances 5 1190 4 838 3 239 1 204 2 127 So my data is unbalanced since 1190 instances are labeled with 5. For the classification Im using scikit's SVC. The problem is I do not know how to balance my data in the right way in order to compute accurately the precision, recall, accuracy and f1-score for the multiclass case. So I tried the following approaches: First: wclf = SVC(kernel='linear', C= 1, class_weight={1: 10}) wclf.fit(X, y) weighted_prediction = wclf.predict(X_test) print 'Accuracy:', accuracy_score(y_test, weighted_prediction) print 'F1 score:', f1_score(y_test, weighted_prediction,average='weighted') print 'Recall:', recall_score(y_test, weighted_prediction, average='weighted') print 'Precision:', precision_score(y_test, weighted_prediction, average='weighted') print '\n clasification report:\n', classification_report(y_test, weighted_prediction) print '\n confussion matrix:\n',confusion_matrix(y_test, weighted_prediction) Second: auto_wclf = SVC(kernel='linear', C= 1, class_weight='auto') auto_wclf.fit(X, y) auto_weighted_prediction = auto_wclf.predict(X_test) print 'Accuracy:', accuracy_score(y_test, auto_weighted_prediction) print 'F1 score:', f1_score(y_test, auto_weighted_prediction, average='weighted') print 'Recall:', recall_score(y_test, auto_weighted_prediction, average='weighted') print 'Precision:', precision_score(y_test, auto_weighted_prediction, average='weighted') print '\n clasification report:\n', classification_report(y_test,auto_weighted_prediction) print '\n confussion matrix:\n',confusion_matrix(y_test, auto_weighted_prediction) Third: clf = SVC(kernel='linear', C= 1) clf.fit(X, y) prediction = clf.predict(X_test) from sklearn.metrics import precision_score, \ recall_score, confusion_matrix, classification_report, \ accuracy_score, f1_score print 'Accuracy:', accuracy_score(y_test, prediction) print 'F1 score:', f1_score(y_test, prediction) print 'Recall:', recall_score(y_test, prediction) print 'Precision:', precision_score(y_test, prediction) print '\n clasification report:\n', classification_report(y_test,prediction) print '\n confussion matrix:\n',confusion_matrix(y_test, prediction) F1 score:/usr/local/lib/python2.7/site-packages/sklearn/metrics/classification.py:676: DeprecationWarning: The default `weighted` averaging is deprecated, and from version 0.18, use of precision, recall or F-score with multiclass or multilabel data or pos_label=None will result in an exception. Please set an explicit value for `average`, one of (None, 'micro', 'macro', 'weighted', 'samples'). In cross validation use, for instance, scoring="f1_weighted" instead of scoring="f1". sample_weight=sample_weight) /usr/local/lib/python2.7/site-packages/sklearn/metrics/classification.py:1172: DeprecationWarning: The default `weighted` averaging is deprecated, and from version 0.18, use of precision, recall or F-score with multiclass or multilabel data or pos_label=None will result in an exception. Please set an explicit value for `average`, one of (None, 'micro', 'macro', 'weighted', 'samples'). In cross validation use, for instance, scoring="f1_weighted" instead of scoring="f1". sample_weight=sample_weight) /usr/local/lib/python2.7/site-packages/sklearn/metrics/classification.py:1082: DeprecationWarning: The default `weighted` averaging is deprecated, and from version 0.18, use of precision, recall or F-score with multiclass or multilabel data or pos_label=None will result in an exception. Please set an explicit value for `average`, one of (None, 'micro', 'macro', 'weighted', 'samples'). In cross validation use, for instance, scoring="f1_weighted" instead of scoring="f1". sample_weight=sample_weight) 0.930416613529 However, Im getting warnings like this: /usr/local/lib/python2.7/site-packages/sklearn/metrics/classification.py:1172: DeprecationWarning: The default `weighted` averaging is deprecated, and from version 0.18, use of precision, recall or F-score with multiclass or multilabel data or pos_label=None will result in an exception. Please set an explicit value for `average`, one of (None, 'micro', 'macro', 'weighted', 'samples'). In cross validation use, for instance, scoring="f1_weighted" instead of scoring="f1" How can I deal correctly with my unbalanced data in order to compute in the right way classifier's metrics?

Output: I think there is a lot of confusion about which weights are used for what. I am not sure I know precisely what bothers you so I am going to cover different topics, bear with me ;). Class weights The weights from the class_weight parameter are used to train the classifier. They are not used in the calculation of any of the metrics you are using: with different class weights, the numbers will be different simply because the classifier is different. Basically in every scikit-learn classifier, the class weights are used to tell your model how important a class is. That means that during the training, the classifier will make extra efforts to classify properly the classes with high weights. How they do that is algorithm-specific. If you want details about how it works for SVC and the doc does not make sense to you, feel free to mention it. The metrics Once you have a classifier, you want to know how well it is performing. Here you can use the metrics you mentioned: accuracy, recall_score, f1_score... Usually when the class distribution is unbalanced, accuracy is considered a poor choice as it gives high scores to models which just predict the most frequent class. I will not detail all these metrics but note that, with the exception of accuracy, they are naturally applied at the class level: as you can see in this print of a classification report they are defined for each class. They rely on concepts such as true positives or false negative that require defining which class is the positive one. precision recall f1-score support 0 0.65 1.00 0.79 17 1 0.57 0.75 0.65 16 2 0.33 0.06 0.10 17 avg / total 0.52 0.60 0.51 50 The warning F1 score:/usr/local/lib/python2.7/site-packages/sklearn/metrics/classification.py:676: DeprecationWarning: The default `weighted` averaging is deprecated, and from version 0.18, use of precision, recall or F-score with multiclass or multilabel data or pos_label=None will result in an exception. Please set an explicit value for `average`, one of (None, 'micro', 'macro', 'weighted', 'samples'). In cross validation use, for instance, scoring="f1_weighted" instead of scoring="f1". You get this warning because you are using the f1-score, recall and precision without defining how they should be computed! The question could be rephrased: from the above classification report, how do you output one global number for the f1-score? You could: Take the average of the f1-score for each class: that's the avg / total result above. It's also called macro averaging. Compute the f1-score using the global count of true positives / false negatives, etc. (you sum the number of true positives / false negatives for each class). Aka micro averaging. Compute a weighted average of the f1-score. Using 'weighted' in scikit-learn will weigh the f1-score by the support of the class: the more elements a class has, the more important the f1-score for this class in the computation. These are 3 of the options in scikit-learn, the warning is there to say you have to pick one. So you have to specify an average argument for the score method. Which one you choose is up to how you want to measure the performance of the classifier: for instance macro-averaging does not take class imbalance into account and the f1-score of class 1 will be just as important as the f1-score of class 5. If you use weighted averaging however you'll get more importance for the class 5. The whole argument specification in these metrics is not super-clear in scikit-learn right now, it will get better in version 0.18 according to the docs. They are removing some non-obvious standard behavior and they are issuing warnings so that developers notice it. Computing scores Last thing I want to mention (feel free to skip it if you're aware of it) is that scores are only meaningful if they are computed on data that the classifier has never seen. This is extremely important as any score you get on data that was used in fitting the classifier is completely irrelevant. Here's a way to do it using StratifiedShuffleSplit, which gives you a random splits of your data (after shuffling) that preserve the label distribution. from sklearn.datasets import make_classification from sklearn.cross_validation import StratifiedShuffleSplit from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score, classification_report, confusion_matrix # We use a utility to generate artificial classification data. X, y = make_classification(n_samples=100, n_informative=10, n_classes=3) sss = StratifiedShuffleSplit(y, n_iter=1, test_size=0.5, random_state=0) for train_idx, test_idx in sss: X_train, X_test, y_train, y_test = X[train_idx], X[test_idx], y[train_idx], y[test_idx] svc.fit(X_train, y_train) y_pred = svc.predict(X_test) print(f1_score(y_test, y_pred, average="macro")) print(precision_score(y_test, y_pred, average="macro")) print(recall_score(y_test, y_pred, average="macro"))


Instruction:
How to display multiple images in one figure

I am trying to display 20 random images on a single Figure. The images are indeed displayed, but they are overlaid. I am using: import numpy as np import matplotlib.pyplot as plt w=10 h=10 fig=plt.figure() for i in range(1,20): img = np.random.randint(10, size=(h,w)) fig.add_subplot(i,2,1) plt.imshow(img) plt.show() I would like them to appear naturally in a grid layout (say 4x5), each with the same size. Part of the problem is that I do not know what the arguments to add_subplot mean. The documentation states that the arguments are the number of rows, number of columns, and plot number. There is no positioning argument. Additionally, the plot number can only be 1 or 2. How can I achieve this?

Output: Here is my approach that you may try: import numpy as np import matplotlib.pyplot as plt w = 10 h = 10 fig = plt.figure(figsize=(8, 8)) columns = 4 rows = 5 for i in range(1, columns*rows +1): img = np.random.randint(10, size=(h,w)) fig.add_subplot(rows, columns, i) plt.imshow(img) plt.show() The resulting image: (Original answer date: Oct 7 '17 at 4:20) Edit 1 Since this answer is popular beyond my expectation. And I see that a small change is needed to enable flexibility for the manipulation of the individual plots. So that I offer this new version to the original code. In essence, it provides:- access to individual axes of subplots possibility to plot more features on selected axes/subplot New code: import numpy as np import matplotlib.pyplot as plt w = 10 h = 10 fig = plt.figure(figsize=(9, 13)) columns = 4 rows = 5 # prep (x,y) for extra plotting xs = np.linspace(0, 2*np.pi, 60) # from 0 to 2pi ys = np.abs(np.sin(xs)) # absolute of sine # ax enables access to manipulate each of subplots ax = [] for i in range(columns*rows): img = np.random.randint(10, size=(h,w)) # create subplot and append to ax ax.append( fig.add_subplot(rows, columns, i+1) ) ax[-1].set_title("ax:"+str(i)) # set title plt.imshow(img, alpha=0.25) # do extra plots on selected axes/subplots # note: index starts with 0 ax[2].plot(xs, 3*ys) ax[19].plot(ys**2, xs) plt.show() # finally, render the plot The resulting plot: Edit 2 In the previous example, the code provides access to the sub-plots with single index, which is inconvenient when the figure has many rows/columns of sub-plots. Here is an alternative of it. The code below provides access to the sub-plots with [row_index][column_index], which is more suitable for manipulation of array of many sub-plots. import matplotlib.pyplot as plt import numpy as np # settings h, w = 10, 10 # for raster image nrows, ncols = 5, 4 # array of sub-plots figsize = [6, 8] # figure size, inches # prep (x,y) for extra plotting on selected sub-plots xs = np.linspace(0, 2*np.pi, 60) # from 0 to 2pi ys = np.abs(np.sin(xs)) # absolute of sine # create figure (fig), and array of axes (ax) fig, ax = plt.subplots(nrows=nrows, ncols=ncols, figsize=figsize) # plot simple raster image on each sub-plot for i, axi in enumerate(ax.flat): # i runs from 0 to (nrows*ncols-1) # axi is equivalent with ax[rowid][colid] img = np.random.randint(10, size=(h,w)) axi.imshow(img, alpha=0.25) # get indices of row/column rowid = i // ncols colid = i % ncols # write row/col indices as axes' title for identification axi.set_title("Row:"+str(rowid)+", Col:"+str(colid)) # one can access the axes by ax[row_id][col_id] # do additional plotting on ax[row_id][col_id] of your choice ax[0][2].plot(xs, 3*ys, color='red', linewidth=3) ax[4][3].plot(ys**2, xs, color='green', linewidth=3) plt.tight_layout(True) plt.show() The resulting plot: Ticks and Tick-labels for Array of Subplots Some of the ticks and tick-labels accompanying the subplots can be hidden to get cleaner plot if all of the subplots share the same value ranges. All of the ticks and tick-labels can be hidden except for the outside edges on the left and bottom like this plot. To achieve the plot with only shared tick-labels on the left and bottom edges, you can do the following:- Add options sharex=True, sharey=True in fig, ax = plt.subplots() That line of code will become: fig,ax=plt.subplots(nrows=nrows,ncols=ncols,figsize=figsize,sharex=True,sharey=True) To specify required number of ticks, and labels to plot, inside the body of for i, axi in enumerate(ax.flat):, add these code axi.xaxis.set_major_locator(plt.MaxNLocator(5)) axi.yaxis.set_major_locator(plt.MaxNLocator(4)) the number 5, and 4 are the number of ticks/tick_labels to plot. You may need other values that suit your plots.


Instruction:
Python os.path.join() on a list

I can do >>> os.path.join("c:/","home","foo","bar","some.txt") 'c:/home\\foo\\bar\\some.txt' But, when I do >>> s = "c:/,home,foo,bar,some.txt".split(",") >>> os.path.join(s) ['c:/', 'home', 'foo', 'bar', 'some.txt'] What am I missing here?

Output: The problem is, os.path.join doesn't take a list as argument, it has to be separate arguments. To unpack the list into separate arguments required by join (and for the record: list was obtained from a string using split), use * - or the 'splat' operator, thus: >>> s = "c:/,home,foo,bar,some.txt".split(",") >>> os.path.join(*s) 'c:/home\\foo\\bar\\some.txt'


Instruction:
Using print statements only to debug

I have been coding a lot in Python of late. And I have been working with data that I haven't worked with before, using formulae never seen before and dealing with huge files. All this made me write a lot of print statements to verify if it's all going right and identify the points of failure. But, generally, outputting so much information is not a good practice. How do I use the print statements only when I want to debug and let them be skipped when I don't want them to be printed?

Output: The logging module in the standard library has everything you could want. It may seem excessive at first, but only use the parts you need. I'd recommend using logging.basicConfig to set the logging level then using the simple log methods: debug, info, warning, error and critical. import logging logging.basicConfig(level=logging.DEBUG) logging.debug('A debug message!') logging.info('We processed %d records', len(processed_records))


Instruction:
Combine f-string and raw string literal

I'm wondering how to use an f-string whilst using r to get a raw string literal. I currently have it as below but would like the option of allowing any name to replace Alex I was thinking adding an f-string and then replacing Alex with curly braces and putting username inside but this doesn't work with the r. username = input('Enter name') download_folder = r'C:\Users\Alex\Downloads'

Output: You can combine the f for an f-string with the r for a raw string: user = 'Alex' dirToSee = fr'C:\Users\{user}\Downloads' print (dirToSee) # prints C:\Users\Alex\Downloads The r only disables backslash escape sequence processing, not f-string processing. Quoting the docs: The 'f' may be combined with 'r', but not with 'b' or 'u', therefore raw formatted strings are possible, but formatted bytes literals are not. ... Unless an 'r' or 'R' prefix is present, escape sequences in string and bytes literals are interpreted...


Instruction:
In python, how do I cast a class object to a dict

Let's say I've got a simple class in python class Wharrgarbl(object): def __init__(self, a, b, c, sum, version='old'): self.a = a self.b = b self.c = c self.sum = 6 self.version = version def __int__(self): return self.sum + 9000 def __what_goes_here__(self): return {'a': self.a, 'b': self.b, 'c': self.c} I can cast it to an integer very easily >>> w = Wharrgarbl('one', 'two', 'three', 6) >>> int(w) 9006 Which is great! But, now I want to cast it to a dict in a similar fashion >>> w = Wharrgarbl('one', 'two', 'three', 6) >>> dict(w) {'a': 'one', 'c': 'three', 'b': 'two'} What do I need to define for this to work? I tried substituting both __dict__ and dict for __what_goes_here__, but dict(w) resulted in a TypeError: Wharrgarbl object is not iterable in both cases. I don't think simply making the class iterable will solve the problem. I also attempted many googles with as many different wordings of "python cast object to dict" as I could think of but couldn't find anything relevant :{ Also! Notice how calling w.__dict__ won't do what I want because it's going to contain w.version and w.sum. I want to customize the cast to dict in the same way that I can customize the cast to int by using def int(self). I know that I could just do something like this >>> w.__what_goes_here__() {'a': 'one', 'c': 'three', 'b': 'two'} But I am assuming there is a pythonic way to make dict(w) work since it is the same type of thing as int(w) or str(w). If there isn't a more pythonic way, that's fine too, just figured I'd ask. Oh! I guess since it matters, this is for python 2.7, but super bonus points for a 2.4 old and busted solution as well. There is another question Overloading __dict__() on python class that is similar to this one but may be different enough to warrant this not being a duplicate. I believe that OP is asking how to cast all the data in his class objects as dictionaries. I'm looking for a more customized approach in that I don't want everything in __dict__ included in the dictionary returned by dict(). Something like public vs private variables may suffice to explain what I'm looking for. The objects will be storing some values used in calculations and such that I don't need/want to show up in the resulting dictionaries. UPDATE: I've chosen to go with the asdict route suggested but it was a tough choice selecting what I wanted to be the answer to the question. Both @RickTeachey and @jpmc26 provided the answer I'm going to roll with but the former had more info and options and landed on the same result as well and was upvoted more so I went with it. Upvotes all around though and thanks for the help. I've lurked long and hard on stackoverflow and I'm trying to get my toes in the water more.

Output: There are at least five six ways. The preferred way depends on what your use case is. Option 1: Simply add an asdict() method. Based on the problem description I would very much consider the asdict way of doing things suggested by other answers. This is because it does not appear that your object is really much of a collection: class Wharrgarbl(object): ... def asdict(self): return {'a': self.a, 'b': self.b, 'c': self.c} Using the other options below could be confusing for others unless it is very obvious exactly which object members would and would not be iterated or specified as key-value pairs. Option 1a: Inherit your class from 'typing.NamedTuple' (or the mostly equivalent 'collections.namedtuple'), and use the _asdict method provided for you. from typing import NamedTuple class Wharrgarbl(NamedTuple): a: str b: str c: str sum: int = 6 version: str = 'old' Using a named tuple is a very convenient way to add lots of functionality to your class with a minimum of effort, including an _asdict method. However, a limitation is that, as shown above, the NT will include all the members in its _asdict. If there are members you don't want to include in your dictionary, you'll need to specify which members you want the named tuple _asdict result to include. To do this, you could either inherit from a base namedtuple class using the older collections.namedtuple API: from collections import namedtuple as nt class Wharrgarbl(nt("Basegarble", "a b c")): # note that the typing info below isn't needed for the old API a: str b: str c: str sum: int = 6 version: str = 'old' ...or you could create a base class using the newer API, and inherit from that, using only the dictionary members in the base class: from typing import NamedTuple class Basegarbl(NamedTuple): a: str b: str c: str class Wharrgarbl(Basegarbl): sum: int = 6 version: str = 'old' Another limitation is that NT is read-only. This may or may not be desirable. Option 2: Implement __iter__. Like this, for example: def __iter__(self): yield 'a', self.a yield 'b', self.b yield 'c', self.c Now you can just do: dict(my_object) This works because the dict() constructor accepts an iterable of (key, value) pairs to construct a dictionary. Before doing this, ask yourself the question whether iterating the object as a series of key,value pairs in this manner- while convenient for creating a dict- might actually be surprising behavior in other contexts. E.g., ask yourself the question "what should the behavior of list(my_object) be...?" Additionally, note that accessing values directly using the get item obj["a"] syntax will not work, and keyword argument unpacking won't work. For those, you'd need to implement the mapping protocol. Option 3: Implement the mapping protocol. This allows access-by-key behavior, casting to a dict without using __iter__, and also provides two types of unpacking behavior: mapping unpacking behavior: {**my_obj} keyword unpacking behavior, but only if all the keys are strings: dict(**my_obj) The mapping protocol requires that you provide (at minimum) two methods together: keys() and __getitem__. class MyKwargUnpackable: def keys(self): return list("abc") def __getitem__(self, key): return dict(zip("abc", "one two three".split()))[key] Now you can do things like: >>> m=MyKwargUnpackable() >>> m["a"] 'one' >>> dict(m) # cast to dict directly {'a': 'one', 'b': 'two', 'c': 'three'} >>> dict(**m) # unpack as kwargs {'a': 'one', 'b': 'two', 'c': 'three'} As mentioned above, if you are using a new enough version of python you can also unpack your mapping-protocol object into a dictionary comprehension like so (and in this case it is not required that your keys be strings): >>> {**m} {'a': 'one', 'b': 'two', 'c': 'three'} Note that the mapping protocol takes precedence over the __iter__ method when casting an object to a dict directly (without using kwarg unpacking, i.e. dict(m)). So it is possible- and might be sometimes convenient- to cause the object to have different behavior when used as an iterable (e.g., list(m)) vs. when cast to a dict (dict(m)). But note also that with regular dictionaries, if you cast to a list, it will give the KEYS back, and not the VALUES as you require. If you implement another nonstandard behavior for __iter__ (returning values instead of keys), it could be surprising for other people using your code unless it is very obvious why this would happen. EMPHASIZED: Just because you CAN use the mapping protocol, does NOT mean that you SHOULD do so. Does it actually make sense for your object to be passed around as a set of key-value pairs, or as keyword arguments and values? Does accessing it by key- just like a dictionary- really make sense? Would you also expect your object to have other standard mapping methods such as items, values, get? Do you want to support the in keyword and equality checks (==)? If the answer to these questions is yes, it's probably a good idea to not stop here, and consider the next option instead. Option 4: Look into using the 'collections.abc' module. Inheriting your class from 'collections.abc.Mapping or 'collections.abc.MutableMapping signals to other users that, for all intents and purposes, your class is a mapping * and can be expected to behave that way. It also provides the methods items, values, get and supports the in keyword and equality checks (==) "for free". You can still cast your object to a dict just as you require, but there would probably be little reason to do so. Because of duck typing, bothering to cast your mapping object to a dict would just be an additional unnecessary step the majority of the time. This answer from me about how to use ABCs might also be helpful. As noted in the comments below: it's worth mentioning that doing this the abc way essentially turns your object class into a dict-like class (assuming you use MutableMapping and not the read-only Mapping base class). Everything you would be able to do with dict, you could do with your own class object. This may be, or may not be, desirable. Also consider looking at the numerical abcs in the numbers module: https://docs.python.org/3/library/numbers.html Since you're also casting your object to an int, it might make more sense to essentially turn your class into a full fledged int so that casting isn't necessary. Option 5: Look into using the dataclasses module (Python 3.7+ only), which includes a convenient asdict() utility method. from dataclasses import dataclass, asdict, field, InitVar @dataclass class Wharrgarbl(object): a: int b: int c: int sum: InitVar[int] # note: InitVar will exclude this from the dict version: InitVar[str] = "old" def __post_init__(self, sum, version): self.sum = 6 # this looks like an OP mistake? self.version = str(version) Now you can do this: >>> asdict(Wharrgarbl(1,2,3,4,"X")) {'a': 1, 'b': 2, 'c': 3} Option 6: Use typing.TypedDict, which has been added in python 3.8. NOTE: option 6 is likely NOT what the OP, or other readers based on the title of this question, are looking for. See additional comments below. class Wharrgarbl(TypedDict): a: str b: str c: str Using this option, the resulting object is a dict (emphasis: the object class will be dict, not Wharrgarbl). There is no reason at all to "cast" it to a dict (unless you are making a copy) because it already is. And since the object is a dict, the initialization signature is identical to that of dict and as such it only accepts keyword arguments or another dictionary. >>> w = Wharrgarbl(a=1,b=2,b=3) >>> w {'a': 1, 'b': 2, 'c': 3} >>> type(w) <class 'dict'> Emphasized: the above "class" Wharrgarbl isn't actually a new class at all. It is simply syntactic sugar for creating typed dict objects with specific keys ONLY and value fields of different types for the type checker. At run time, it is still nothing more than a dict. This option can be pretty convenient for signaling to readers of your code (and also to a type checker such as mypy) that these particular dict objects are expected to have specific keys with specific value types. And it also will allow code completion tools, such as Pycharm or the VSCode python extension, to do their magic. But this means you cannot, for example, add other methods, although you can try: class MyDict(TypedDict): def my_fancy_method(self): return "world changing result" ...but it won't work: >>> MyDict().my_fancy_method() Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'dict' object has no attribute 'my_fancy_method' * "Mapping" has become the standard "name" of the dict-like duck type


Instruction:
How to throw error and exit with a custom message in python

I've seen people suggesting sys.exit() in Python. My question is that, is there any other way to exit the execution of current script, I mean termination, with an error. Something like this: sys.exit("You can not have three process at the same time.") Currently my solution would be: print("You can not have three process at the same time.") sys.exit()

Output: Calling sys.exit with a string will work. The docs mention this use explicitly: In particular, sys.exit("some error message") is a quick way to exit a program when an error occurs. To be more specific this will also result in an exit code of 1: any other object is printed to stderr and results in an exit code of 1


Instruction:
Call Python script from bash with argument

I know that I can run a python script from my bash script using the following: python python_script.py But what about if I wanted to pass a variable / argument to my python script from my bash script. How can I do that? Basically bash will work out a filename and then python will upload it, but I need to send the filename from bash to python when I call it.

Output: To execute a python script in a bash script you need to call the same command that you would within a terminal. For instance > python python_script.py var1 var2 To access these variables within python you will need import sys print(sys.argv[0]) # prints python_script.py print(sys.argv[1]) # prints var1 print(sys.argv[2]) # prints var2


Instruction:
Skip rows during csv import pandas

I'm trying to import a .csv file using pandas.read_csv(), however, I don't want to import the 2nd row of the data file (the row with index = 1 for 0-indexing). I can't see how not to import it because the arguments used with the command seem ambiguous: From the pandas website: skiprows : list-like or integer Row numbers to skip (0-indexed) or number of rows to skip (int) at the start of the file." If I put skiprows=1 in the arguments, how does it know whether to skip the first row or skip the row with index 1?

Output: You can try yourself: >>> import pandas as pd >>> from io import StringIO >>> s = """1, 2 ... 3, 4 ... 5, 6""" >>> pd.read_csv(StringIO(s), skiprows=[1], header=None) 0 1 0 1 2 1 5 6 >>> pd.read_csv(StringIO(s), skiprows=1, header=None) 0 1 0 3 4 1 5 6


Instruction:
Writelines writes lines without newline, Just fills the file

I have a program that writes a list to a file. The list is a list of pipe delimited lines and the lines should be written to the file like this: 123|GSV|Weather_Mean|hello|joe|43.45 122|GEV|temp_Mean|hello|joe|23.45 124|GSI|Weather_Mean|hello|Mike|47.45 BUT it wrote them line this ahhhh: 123|GSV|Weather_Mean|hello|joe|43.45122|GEV|temp_Mean|hello|joe|23.45124|GSI|Weather_Mean|hello|Mike|47.45 This program wrote all the lines into like one line without any line breaks.. This hurts me a lot and I gotta figure-out how to reverse this but anyway, where is my program wrong here? I thought write lines should write lines down the file rather than just write everything to one line.. fr = open(sys.argv[1], 'r') # source file fw = open(sys.argv[2]+"/masked_"+sys.argv[1], 'w') # Target Directory Location for line in fr: line = line.strip() if line == "": continue columns = line.strip().split('|') if columns[0].find("@") > 1: looking_for = columns[0] # this is what we need to search else: looking_for = "Dummy@dummy.com" if looking_for in d: # by default, iterating over a dictionary will return keys new_line = d[looking_for]+'|'+'|'.join(columns[1:]) line_list.append(new_line) else: new_idx = str(len(d)+1) d[looking_for] = new_idx kv = open(sys.argv[3], 'a') kv.write(looking_for+" "+new_idx+'\n') kv.close() new_line = d[looking_for]+'|'+'|'.join(columns[1:]) line_list.append(new_line) fw.writelines(line_list)

Output: This is actually a pretty common problem for newcomers to Python—especially since, across the standard library and popular third-party libraries, some reading functions strip out newlines, but almost no writing functions (except the log-related stuff) add them. So, there's a lot of Python code out there that does things like: fw.write('\n'.join(line_list) + '\n') (writing a single string) or fw.writelines(line + '\n' for line in line_list) Either one is correct, and of course you could even write your own writelinesWithNewlines function that wraps it up… But you should only do this if you can't avoid it. It's better if you can create/keep the newlines in the first place—as in Greg Hewgill's suggestions: line_list.append(new_line + "\n") And it's even better if you can work at a higher level than raw lines of text, e.g., by using the csv module in the standard library, as esuaro suggests. For example, right after defining fw, you might do this: cw = csv.writer(fw, delimiter='|') Then, instead of this: new_line = d[looking_for]+'|'+'|'.join(columns[1:]) line_list.append(new_line) You do this: row_list.append(d[looking_for] + columns[1:]) And at the end, instead of this: fw.writelines(line_list) You do this: cw.writerows(row_list) Finally, your design is "open a file, then build up a list of lines to add to the file, then write them all at once". If you're going to open the file up top, why not just write the lines one by one? Whether you're using simple writes or a csv.writer, it'll make your life simpler, and your code easier to read. (Sometimes there can be simplicity, efficiency, or correctness reasons to write a file all at once—but once you've moved the open all the way to the opposite end of the program from the write, you've pretty much lost any benefits of all-at-once.)


Instruction:
Calling Python in Java?

I am wondering if it is possible to call Python functions from Java code using Jython, or is it only for calling Java code from Python?

Output: Jython: Python for the Java Platform - http://www.jython.org/index.html You can easily call python functions from Java code with Jython. That is as long as your python code itself runs under jython, i.e. doesn't use some c-extensions that aren't supported. If that works for you, it's certainly the simplest solution you can get. Otherwise you can use org.python.util.PythonInterpreter from the new Java6 interpreter support. A simple example from the top of my head - but should work I hope: (no error checking done for brevity) PythonInterpreter interpreter = new PythonInterpreter(); interpreter.exec("import sys\nsys.path.append('pathToModules if they are not there by default')\nimport yourModule"); // execute a function that takes a string and returns a string PyObject someFunc = interpreter.get("funcName"); PyObject result = someFunc.__call__(new PyString("Test!")); String realResult = (String) result.__tojava__(String.class); As of 2021, Jython does not support Python 3.x


Instruction:
What shebang to use for Python scripts run under a pyenv virtualenv

When a Python script is supposed to be run from a pyenv virtualenv, what is the correct shebang for the file? As an example test case, the default Python on my system (OS X) does not have pandas installed. The pyenv virtualenv venv_name does. I tried getting the path of the Python executable from the virtualenv. pyenv activate venv_name which python Output: /Users/username/.pyenv/shims/python So I made my example script.py: #!/Users/username/.pyenv/shims/python import pandas as pd print 'success' But when I tried running the script (from within 'venv_name'), I got an error: ./script.py Output: ./script.py: line 2: import: command not found ./script.py: line 3: print: command not found Although running that path directly on the command line (from within 'venv_name') works fine: /Users/username/.pyenv/shims/python script.py Output: success And: python script.py # Also works Output: success What is the proper shebang for this? Ideally, I want something generic so that it will point at the Python of whatever my current venv is.

Output: I don't really know why calling the interpreter with the full path wouldn't work for you. I use it all the time. But if you want to use the Python interpreter that is in your environment, you should do: #!/usr/bin/env python That way you search your environment for the Python interpreter to use.


Instruction:
What is print(f"...")

I am reading through a python script that takes an input of XML files and outputs an XML file. However, I do not understand the printing syntax. Can someone please explain what f in print(f"...") does? args = parser.parser_args() print(f"Input directory: {args.input_directory}") print(f"Output directory: {args.output_directory}")

Output: The f means Formatted string literals and it's new in Python 3.6. A formatted string literal or f-string is a string literal that is prefixed with f or F. These strings may contain replacement fields, which are expressions delimited by curly braces {}. While other string literals always have a constant value, formatted strings are really expressions evaluated at run time. Some examples of formatted string literals: >>> name = "Fred" >>> f"He said his name is {name}." "He said his name is Fred." >>> name = "Fred" >>> f"He said his name is {name!r}." "He said his name is Fred." >>> f"He said his name is {repr(name)}." # repr() is equivalent to !r "He said his name is Fred." >>> width = 10 >>> precision = 4 >>> value = decimal.Decimal("12.34567") >>> f"result: {value:{width}.{precision}}" # nested fields result: 12.35 >>> today = datetime(year=2023, month=1, day=27) >>> f"{today:%B %d, %Y}" # using date format specifier January 27, 2023 >>> number = 1024 >>> f"{number:#0x}" # using integer format specifier 0x400


Instruction:
How to fix AttributeError: partially initialized module?

I am trying to run my script but keep getting this error: File ".\checkmypass.py", line 1, in <module> import requests line 3, in <module> response = requests.get(url) AttributeError: partially initialized module 'requests' has no attribute 'get' (most likely due to a circular import) How can I fix it?

Output: This can happen when there's a local file with the same name as an imported module – Python sees the local file and thinks it's the module. In my case, I had a file I created in the same folder called requests.py. So my code was actually importing that file and not the actual requests module you install with pip. Then I had another issue with a file I created called logging.py. I renamed both files and the issue was resolved.


Instruction:
Convert PIL Image to byte array?

I have an image in PIL Image format. I need to convert it to byte array. img = Image.open(fh, mode='r') roiImg = img.crop(box) Now I need the roiImg as a byte array.

Output: import io from PIL import Image img = Image.open(fh, mode='r') roi_img = img.crop(box) img_byte_arr = io.BytesIO() roi_img.save(img_byte_arr, format='PNG') img_byte_arr = img_byte_arr.getvalue() With this, I don't have to save the cropped image on my disk and I am able to retrieve the byte array from a PIL cropped image.


Instruction:
How can I reorder a list?

Given an arbitrary array of size n, I'd like to reorganize the elements of the array based on the array's discrete indices. Python example: # Unique array of size n [ "a", "b", "c", "d", "e", ... <n> ] # Indices of array [ 0, 1, 2, 3, 4, ... <index_of_n> ] # Desired re-organization function 'indexMove' indexMove( [ "a", "b", "c", "d", "e", ... <n> ], [ <index_of_n>, 4, 0, 2, 3, ... 1 ] ) # Desired output from indexMove operation [ <n>, "e", "a", "c", "d", ... "b" ] What is the fastest way to perform this operation (achieving the smallest time complexity)?

Output: You can do it like this mylist = ['a', 'b', 'c', 'd', 'e'] myorder = [3, 2, 0, 1, 4] mylist = [mylist[i] for i in myorder] print(mylist) # prints: ['d', 'c', 'a', 'b', 'e']


Instruction:
Count unique values using pandas groupby

I have data of the following form: df = pd.DataFrame({ 'group': [1, 1, 2, 3, 3, 3, 4], 'param': ['a', 'a', 'b', np.nan, 'a', 'a', np.nan] }) print(df) # group param # 0 1 a # 1 1 a # 2 2 b # 3 3 NaN # 4 3 a # 5 3 a # 6 4 NaN Non-null values within groups are always the same. I want to count the non-null value for each group (where it exists) once, and then find the total counts for each value. I'm currently doing this in the following (clunky and inefficient) way: param = [] for _, group in df[df.param.notnull()].groupby('group'): param.append(group.param.unique()[0]) print(pd.DataFrame({'param': param}).param.value_counts()) # a 2 # b 1 I'm sure there's a way to do this more cleanly and without using a loop, but I just can't seem to work it out. Any help would be much appreciated.

Output: I think you can use SeriesGroupBy.nunique: print (df.groupby('param')['group'].nunique()) param a 2 b 1 Name: group, dtype: int64 Another solution with unique, then create new df by DataFrame.from_records, reshape to Series by stack and last value_counts: a = df[df.param.notnull()].groupby('group')['param'].unique() print (pd.DataFrame.from_records(a.values.tolist()).stack().value_counts()) a 2 b 1 dtype: int64


Instruction:
Why doesn't requests.get() return? What is the default timeout that requests.get() uses?

In my script, requests.get never returns: import requests print ("requesting..") # This call never returns! r = requests.get( "http://www.some-site.example", proxies = {'http': '222.255.169.74:8080'}, ) print(r.ok) What could be the possible reason(s)? Any remedy? What is the default timeout that get uses?

Output: What is the default timeout that get uses? The default timeout is None, which means it'll wait (hang) until the connection is closed. Just specify a timeout value, like this: r = requests.get( 'http://www.example.com', proxies={'http': '222.255.169.74:8080'}, timeout=5 )


Instruction:
How does a generator comprehension works?

What does generator comprehension do? How does it work? I couldn't find a tutorial about it.

Output: Do you understand list comprehensions? If so, a generator expression is like a list comprehension, but instead of finding all the items you're interested and packing them into list, it waits, and yields each item out of the expression, one by one. >>> my_list = [1, 3, 5, 9, 2, 6] >>> filtered_list = [item for item in my_list if item > 3] >>> print(filtered_list) [5, 9, 6] >>> len(filtered_list) 3 >>> # compare to generator expression ... >>> filtered_gen = (item for item in my_list if item > 3) >>> print(filtered_gen) # notice it's a generator object <generator object <genexpr> at 0x7f2ad75f89e0> >>> len(filtered_gen) # So technically, it has no length Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: object of type 'generator' has no len() >>> # We extract each item out individually. We'll do it manually first. ... >>> next(filtered_gen) 5 >>> next(filtered_gen) 9 >>> next(filtered_gen) 6 >>> next(filtered_gen) # Should be all out of items and give an error Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration >>> # Yup, the generator is spent. No values for you! ... >>> # Let's prove it gives the same results as our list comprehension ... >>> filtered_gen = (item for item in my_list if item > 3) >>> gen_to_list = list(filtered_gen) >>> print(gen_to_list) [5, 9, 6] >>> filtered_list == gen_to_list True >>> Because a generator expression only has to yield one item at a time, it can lead to big savings in memory usage. Generator expressions make the most sense in scenarios where you need to take one item at a time, do a lot of calculations based on that item, and then move on to the next item. If you need more than one value, you can also use a generator expression and grab a few at a time. If you need all the values before your program proceeds, use a list comprehension instead.


Instruction:
Filtering Pandas Dataframe using OR statement

I have a pandas dataframe and I want to filter the whole df based on the value of two columns in the data frame. I want to get back all rows and columns where IBRD or IMF != 0. alldata_balance = alldata[(alldata[IBRD] !=0) or (alldata[IMF] !=0)] but this gives me a ValueError ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all(). So I know I am not using the or statement correctly, is there a way to do this?

Output: From the docs: Another common operation is the use of boolean vectors to filter the data. The operators are: | for or, & for and, and ~ for not. These must be grouped by using parentheses. https://pandas.pydata.org/docs/user_guide/indexing.html#boolean-indexing Try: alldata_balance = alldata[(alldata[IBRD] !=0) | (alldata[IMF] !=0)]


Instruction:
When should I use hstack/vstack vs append vs concatenate vs column_stack?

Simple question: what is the advantage of each of these methods. It seems that given the right parameters (and ndarray shapes) they all work seemingly equivalently. Do some work in place? Have better performance? Which functions should I use when?

Output: All the functions are written in Python except np.concatenate. With an IPython shell you just use ??. If not, here's a summary of their code: vstack concatenate([atleast_2d(_m) for _m in tup], 0) i.e. turn all inputs in to 2d (or more) and concatenate on first hstack concatenate([atleast_1d(_m) for _m in tup], axis=<0 or 1>) colstack transform arrays with (if needed) array(arr, copy=False, subok=True, ndmin=2).T append concatenate((asarray(arr), values), axis=axis) In other words, they all work by tweaking the dimensions of the input arrays, and then concatenating on the right axis. They are just convenience functions. And newer np.stack: arrays = [asanyarray(arr) for arr in arrays] shapes = set(arr.shape for arr in arrays) result_ndim = arrays[0].ndim + 1 axis = normalize_axis_index(axis, result_ndim) sl = (slice(None),) * axis + (_nx.newaxis,) expanded_arrays = [arr[sl] for arr in arrays] concatenate(expanded_arrays, axis=axis, out=out) That is, it expands the dims of all inputs (a bit like np.expand_dims), and then concatenates. With axis=0, the effect is the same as np.array. hstack documentation now adds: The functions concatenate, stack and block provide more general stacking and concatenation operations. np.block is also new. It, in effect, recursively concatenates along the nested lists.


Instruction:
DistutilsOptionError: must supply either home or prefix/exec-prefix -- not both

I've been usually installed python packages through pip. For Google App Engine, I need to install packages to another target directory. I've tried: pip install -I flask-restful --target ./lib but it fails with: must supply either home or prefix/exec-prefix -- not both How can I get this to work?

Output: Are you using OS X and Homebrew? The Homebrew python page https://github.com/Homebrew/brew/blob/master/docs/Homebrew-and-Python.md calls out a known issue with pip and a work around. Worked for me. You can make this "empty prefix" the default by adding a ~/.pydistutils.cfg file with the following contents: [install] prefix= Edit: The Homebrew page was later changed to recommend passing --prefix on the command line, as discussed in the comments below. Here is the last version which contained that text. Unfortunately this only works for sdists, not wheels. The issue was reported to pip, which later fixed it for --user. That's probably why the section has now been removed from the Homebrew page. However, the problem still occurs when using --target as in the question above.