Instruction:
Scatter plot with different text at each data point

I am trying to make a scatter plot and annotate data points with different numbers from a list. So, for example, I want to plot y vs x and annotate with corresponding numbers from n. y = [2.56422, 3.77284, 3.52623, 3.51468, 3.02199] x = [0.15, 0.3, 0.45, 0.6, 0.75] n = [58, 651, 393, 203, 123] ax = fig.add_subplot(111) ax1.scatter(z, y, fmt='o') Any ideas?

Output: I'm not aware of any plotting method which takes arrays or lists but you could use annotate() while iterating over the values in n. import matplotlib.pyplot as plt x = [0.15, 0.3, 0.45, 0.6, 0.75] y = [2.56422, 3.77284, 3.52623, 3.51468, 3.02199] n = [58, 651, 393, 203, 123] fig, ax = plt.subplots() ax.scatter(x, y) for i, txt in enumerate(n): ax.annotate(txt, (x[i], y[i])) There are a lot of formatting options for annotate(), see the matplotlib website:


Instruction:
How do I compare version numbers in Python?

I am walking a directory that contains eggs to add those eggs to the sys.path. If there are two versions of the same .egg in the directory, I want to add only the latest one. I have a regular expression r"^(?P<eggName>\w+)-(?P<eggVersion>[\d\.]+)-.+\.egg$ to extract the name and version from the filename. The problem is comparing the version number, which is a string like 2.3.1. Since I'm comparing strings, 2 sorts above 10, but that's not correct for versions. >>> "2.3.1" > "10.1.1" True I could do some splitting, parsing, casting to int, etc., and I would eventually get a workaround. But this is Python, not Java. Is there an elegant way to compare version strings?

Output: Use packaging.version.Version which supports PEP 440 style ordering of version strings. >>> # pip install packaging >>> from packaging.version import Version >>> Version("2.3.1") < Version("10.1.2") True >>> Version("1.3.a4") < Version("10.1.2") True An ancient and now deprecated method you might encounter is distutils.version, it's undocumented and conforms only to the superseded PEP 386; >>> from distutils.version import LooseVersion, StrictVersion >>> LooseVersion("2.3.1") < LooseVersion("10.1.2") True >>> StrictVersion("2.3.1") < StrictVersion("10.1.2") True >>> StrictVersion("1.3.a4") Traceback (most recent call last): ... ValueError: invalid version number '1.3.a4' As you can see it sees valid PEP 440 versions as “not strict” and therefore doesn’t match modern Python’s notion of what a valid version is. As distutils.version is undocumented, here are the relevant docstrings.


Instruction:
What alternative is there to execfile in Python 3? / How to include a Python file?

It seems like in Python 3 they've removed all of the easy ways to quickly load a script, by removing execfile(). What alternative is there to include a Python file in another one, and execute it?

Output: According to the Python documentation, instead of this: execfile("./filename") Use this: exec(open("./filename").read()) See Python's docs for: What’s New In Python 3.0 execfile exec


Instruction:
What is the difference between flatten and ravel functions in numpy?

import numpy as np y = np.array(((1,2,3),(4,5,6),(7,8,9))) OUTPUT: print(y.flatten()) [1 2 3 4 5 6 7 8 9] print(y.ravel()) [1 2 3 4 5 6 7 8 9] Both function return the same list. Then what is the need of two different functions performing same job.

Output: The current API is that: flatten always returns a copy. ravel returns a contiguous view of the original array whenever possible. This isn't visible in the printed output, but if you modify the array returned by ravel, it may modify the entries in the original array. If you modify the entries in an array returned from flatten this will never happen. ravel will often be faster since no memory is copied, but you have to be more careful about modifying the array it returns. reshape((-1,)) gets a view whenever the strides of the array allow it even if that means you don't always get a contiguous array.


Instruction:
How can I create an object and add attributes to it?

I want to create a dynamic object in Python and then add attributes to it. This didn't work: obj = object() obj.somefield = "somevalue" AttributeError: 'object' object has no attribute 'somefield' For details on why it doesn't work, see Can't set attributes on instance of "object" class.

Output: You could use my ancient Bunch recipe, but if you don't want to make a "bunch class", a very simple one already exists in Python -- all functions can have arbitrary attributes (including lambda functions). So, the following works: obj = lambda: None obj.somefield = 'somevalue' Whether the loss of clarity compared to the venerable Bunch recipe is OK, is a style decision I will of course leave up to you.


Instruction:
How do I create an empty array and then append to it in NumPy?

I want to create an empty array and append items to it, one at a time. xs = [] for item in data: xs.append(item) Can I use this list-style notation with NumPy arrays?

Output: That is the wrong mental model for using NumPy efficiently. NumPy arrays are stored in contiguous blocks of memory. To append rows or columns to an existing array, the entire array needs to be copied to a new block of memory, creating gaps for the new elements to be stored. This is very inefficient if done repeatedly. Instead of appending rows, allocate a suitably sized array, and then assign to it row-by-row: >>> import numpy as np >>> a = np.zeros(shape=(3, 2)) >>> a array([[ 0., 0.], [ 0., 0.], [ 0., 0.]]) >>> a[0] = [1, 2] >>> a[1] = [3, 4] >>> a[2] = [5, 6] >>> a array([[ 1., 2.], [ 3., 4.], [ 5., 6.]])


Instruction:
Is __init__.py not required for packages in Python 3.3+

I am using Python 3.5.1. I read the document and the package section here: https://docs.python.org/3/tutorial/modules.html#packages Now, I have the following structure: /home/wujek/Playground/a/b/module.py module.py: class Foo: def __init__(self): print('initializing Foo') Now, while in /home/wujek/Playground: ~/Playground $ python3 >>> import a.b.module >>> a.b.module.Foo() initializing Foo <a.b.module.Foo object at 0x100a8f0b8> Similarly, now in home, superfolder of Playground: ~ $ PYTHONPATH=Playground python3 >>> import a.b.module >>> a.b.module.Foo() initializing Foo <a.b.module.Foo object at 0x10a5fee10> Actually, I can do all kinds of stuff: ~ $ PYTHONPATH=Playground python3 >>> import a >>> import a.b >>> import Playground.a.b Why does this work? I though there needed to be __init__.py files (empty ones would work) in both a and b for module.py to be importable when the Python path points to the Playground folder? This seems to have changed from Python 2.7: ~ $ PYTHONPATH=Playground python >>> import a ImportError: No module named a >>> import a.b ImportError: No module named a.b >>> import a.b.module ImportError: No module named a.b.module With __init__.py in both ~/Playground/a and ~/Playground/a/b it works fine.

Output: Overview @Mike's answer is correct but too imprecise. It is true that Python 3.3+ supports Implicit Namespace Packages that allows it to create a package without an __init__.py file. This is called a namespace package in contrast to a regular package which does have an __init__.py file (empty or not empty). However, creating a namespace package should ONLY be done if there is a need for it. For most use cases and developers out there, this doesn't apply so you should stick with EMPTY __init__.py files regardless. Namespace package use case To demonstrate the difference between the two types of python packages, lets look at the following example: google_pubsub/ <- Package 1 google/ <- Namespace package (there is no __init__.py) cloud/ <- Namespace package (there is no __init__.py) pubsub/ <- Regular package (with __init__.py) __init__.py <- Required to make the package a regular package foo.py google_storage/ <- Package 2 google/ <- Namespace package (there is no __init__.py) cloud/ <- Namespace package (there is no __init__.py) storage/ <- Regular package (with __init__.py) __init__.py <- Required to make the package a regular package bar.py google_pubsub and google_storage are separate packages but they share the same namespace google/cloud. In order to share the same namespace, it is required to make each directory of the common path a namespace package, i.e. google/ and cloud/. This should be the only use case for creating namespace packages, otherwise, there is no need for it. It's crucial that there are no __init__py files in the google and google/cloud directories so that both directories can be interpreted as namespace packages. In Python 3.3+ any directory on the sys.path with a name that matches the package name being looked for will be recognized as contributing modules and subpackages to that package. As a result, when you import both from google_pubsub and google_storage, the Python interpreter will be able to find them. This is different from regular packages which are self-contained meaning all parts live in the same directory hierarchy. When importing a package and the Python interpreter encounters a subdirectory on the sys.path with an __init__.py file, then it will create a single directory package containing only modules from that directory, rather than finding all appropriately named subdirectories outside that directory. This is perfectly fine for packages that don't want to share a namespace. I highly recommend taking a look at Traps for the Unwary in Python’s Import System to get a better understanding of how Python importing behaves with regular and namespace package and what __init__.py traps to watch out for. Summary Only skip __init__.py files if you want to create namespace packages. Only create namespace packages if you have different libraries that reside in different locations and you want them each to contribute a subpackage to the parent package, i.e. the namespace package. Keep on adding empty __init__.py to your directories because 99% of the time you just want to create regular packages. Also, Python tools out there such as mypy and pytest require empty __init__.py files to interpret the code structure accordingly. This can lead to weird errors if not done with care. Resources My answer only touches the surface of how regular packages and namespace packages work, so take a look at the following resources for further information: PEP 420 -- Implicit Namespace Packages The import system - Regular packages The import system - Namespace packages Traps for the Unwary in Python’s Import System


Instruction:
Reading binary file and looping over each byte

In Python, how do I read in a binary file and loop over each byte of that file?

Output: Python >= 3.8 Thanks to the walrus operator (:=) the solution is quite short. We read bytes objects from the file and assign them to the variable byte with open("myfile", "rb") as f: while (byte := f.read(1)): # Do stuff with byte. Python >= 3 In older Python 3 versions, we get have to use a slightly more verbose way: with open("myfile", "rb") as f: byte = f.read(1) while byte != b"": # Do stuff with byte. byte = f.read(1) Or as benhoyt says, skip the not equal and take advantage of the fact that b"" evaluates to false. This makes the code compatible between 2.6 and 3.x without any changes. It would also save you from changing the condition if you go from byte mode to text or the reverse. with open("myfile", "rb") as f: byte = f.read(1) while byte: # Do stuff with byte. byte = f.read(1) Python >= 2.5 In Python 2, it's a bit different. Here we don't get bytes objects, but raw characters: with open("myfile", "rb") as f: byte = f.read(1) while byte != "": # Do stuff with byte. byte = f.read(1) Note that the with statement is not available in versions of Python below 2.5. To use it in v 2.5 you'll need to import it: from __future__ import with_statement In 2.6 this is not needed. Python 2.4 and Earlier f = open("myfile", "rb") try: byte = f.read(1) while byte != "": # Do stuff with byte. byte = f.read(1) finally: f.close()


Instruction:
Python subprocess/Popen with a modified environment

I believe that running an external command with a slightly modified environment is a very common case. That's how I tend to do it: import subprocess, os my_env = os.environ my_env["PATH"] = "/usr/sbin:/sbin:" + my_env["PATH"] subprocess.Popen(my_command, env=my_env) I've got a gut feeling that there's a better way; does it look alright?

Output: I think os.environ.copy() is better if you don't intend to modify the os.environ for the current process: import subprocess, os my_env = os.environ.copy() my_env["PATH"] = f"/usr/sbin:/sbin:{my_env['PATH']}" subprocess.Popen(my_command, env=my_env)


Instruction:
How to get last items of a list in Python?

I need the last 9 numbers of a list and I'm sure there is a way to do it with slicing, but I can't seem to get it. I can get the first 9 like this: num_list[0:9]

Output: You can use negative integers with the slicing operator for that. Here's an example using the python CLI interpreter: >>> a = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] >>> a[-9:] [4, 5, 6, 7, 8, 9, 10, 11, 12] the important line is a[-9:]


Instruction:
Python setup.py develop vs install

Two options in setup.py develop and install are confusing me. According to this site, using develop creates a special link to site-packages directory. People have suggested that I use python setup.py install for a fresh installation and python setup.py develop after any changes have been made to the setup file. Can anyone shed some light on the usage of these commands?

Output: python setup.py install is used to install (typically third party) packages that you're not going to develop/modify/debug yourself. For your own stuff, you want to first install your package and then be able to frequently edit the code without having to re-install the package every time — and that is exactly what python setup.py develop does: it installs the package (typically just a source folder) in a way that allows you to conveniently edit your code after it’s installed to the (virtual) environment, and have the changes take effect immediately. Note: It is highly recommended to use pip install . (regular install) and pip install -e . (developer install) to install packages, as invoking setup.py directly will do the wrong things for many dependencies, such as pull prereleases and incompatible package versions, or make the package hard to uninstall with pip. Update: The develop counterpart for the latest python -m build approach is as follows (as per):


Instruction:
Reference requirements.txt for the install_requires kwarg in setuptools setup.py file

I have a requirements.txt file that I'm using with Travis-CI. It seems silly to duplicate the requirements in both requirements.txt and setup.py, so I was hoping to pass a file handle to the install_requires kwarg in setuptools.setup. Is this possible? If so, how should I go about doing it? Here is my requirements.txt file: guessit>=0.5.2 tvdb_api>=1.8.2 hachoir-metadata>=1.3.3 hachoir-core>=1.3.3 hachoir-parser>=1.3.4

Output: Update 12/2024: This does not work in recent versions of Python, primarily because the function has been moved around in the pip module. Either way, as stated below by maintainers of the library, it is not recommended to import and use this function, as it is an internal function which is subject to change and being moved around. Old answer: You can flip it around and list the dependencies in setup.py and have a single character — a dot . — in requirements.txt instead. Alternatively, even if not advised, it is still possible to parse the requirements.txt file (if it doesn't refer any external requirements by URL) with the following hack (tested with pip 9.0.1): install_reqs = parse_requirements('requirements.txt', session='hack') This doesn't filter environment markers though. In old versions of pip, more specifically older than 6.0, there is a public API that can be used to achieve this. A requirement file can contain comments (#) and can include some other files (--requirement or -r). Thus, if you really want to parse a requirements.txt you can use the pip parser: from pip.req import parse_requirements # parse_requirements() returns generator of pip.req.InstallRequirement objects install_reqs = parse_requirements(<requirements_path>) # reqs is a list of requirement # e.g. ['django==1.5.1', 'mezzanine==1.4.6'] reqs = [str(ir.req) for ir in install_reqs] setup( ... install_requires=reqs )


Instruction:
Numpy array dimensions

How do I get the dimensions of an array? For instance, this is 2x2: a = np.array([[1, 2], [3, 4]])

Output: Use .shape to obtain a tuple of array dimensions: >>> a.shape (2, 2)


Instruction:
Is there a built in function for string natural sort?

I have a list of strings for which I would like to perform a natural alphabetical sort. For instance, the following list is naturally sorted (what I want): ['elm0', 'elm1', 'Elm2', 'elm9', 'elm10', 'Elm11', 'Elm12', 'elm13'] And here's the "sorted" version of the above list (what I get using sorted()): ['Elm11', 'Elm12', 'Elm2', 'elm0', 'elm1', 'elm10', 'elm13', 'elm9'] I'm looking for a sort function which behaves like the first one.

Output: There is a third party library for this on PyPI called natsort (full disclosure, I am the package's author). For your case, you can do either of the following: >>> from natsort import natsorted, ns >>> x = ['Elm11', 'Elm12', 'Elm2', 'elm0', 'elm1', 'elm10', 'elm13', 'elm9'] >>> natsorted(x, key=lambda y: y.lower()) ['elm0', 'elm1', 'Elm2', 'elm9', 'elm10', 'Elm11', 'Elm12', 'elm13'] >>> natsorted(x, alg=ns.IGNORECASE) # or alg=ns.IC ['elm0', 'elm1', 'Elm2', 'elm9', 'elm10', 'Elm11', 'Elm12', 'elm13'] You should note that natsort uses a general algorithm so it should work for just about any input that you throw at it. If you want more details on why you might choose a library to do this rather than rolling your own function, check out the natsort documentation's How It Works page, in particular the Special Cases Everywhere! section. If you need a sorting key instead of a sorting function, use either of the below formulas. >>> from natsort import natsort_keygen, ns >>> l1 = ['elm0', 'elm1', 'Elm2', 'elm9', 'elm10', 'Elm11', 'Elm12', 'elm13'] >>> l2 = l1[:] >>> natsort_key1 = natsort_keygen(key=lambda y: y.lower()) >>> l1.sort(key=natsort_key1) >>> l1 ['elm0', 'elm1', 'Elm2', 'elm9', 'elm10', 'Elm11', 'Elm12', 'elm13'] >>> natsort_key2 = natsort_keygen(alg=ns.IGNORECASE) >>> l2.sort(key=natsort_key2) >>> l2 ['elm0', 'elm1', 'Elm2', 'elm9', 'elm10', 'Elm11', 'Elm12', 'elm13'] Update November 2020 Given that a popular request/question is "how to sort like Windows Explorer?" (or whatever is your operating system's file system browser), as of natsort version 7.1.0 there is a function called os_sorted to do exactly this. On Windows, it will sort in the same order as Windows Explorer, and on other operating systems it should sort like whatever is the local file system browser. >>> from natsort import os_sorted >>> os_sorted(list_of_paths) # your paths sorted like your file system browser For those needing a sort key, you can use os_sort_keygen (or os_sort_key if you just need the defaults). Caveat - Please read the API documentation for this function before you use to understand the limitations and how to get best results.


Instruction:
How to check if a variable is a dictionary in Python?

How would you check if a variable is a dictionary in Python? For example, I'd like it to loop through the values in the dictionary until it finds a dictionary. Then, loop through the one it finds: dict = {'abc': 'abc', 'def': {'ghi': 'ghi', 'jkl': 'jkl'}} for k, v in dict.items(): if ###check if v is a dictionary: for k, v in v.iteritems(): print(k, ' ', v) else: print(k, ' ', v)

Output: Use isinstance(ele, dict), which will return true for dict objects as well as subclasses of dict, such as OrderedDict and defaultdict: d = {'abc': 'abc', 'def': {'ghi': 'ghi', 'jkl': 'jkl'}} for element in d.values(): if isinstance(element, dict): for k, v in element.items(): print(k, ' ', v) You can do if type(ele) is dict if you want to check strictly for instances of dict and not subclasses of it, though this is generally not recommended.


Instruction:
How to write a Python module/package?

I've been making Python scripts for simple tasks at work and never really bothered packaging them for others to use. Now I have been assigned to make a Python wrapper for a REST API. I have absolutely no idea on how to start and I need help. What I have: (Just want to be specific as possible) I have the virtualenv ready, it's also up in github, the .gitignore file for python is there as well, plus, the requests library for interacting with the REST API. That's it. Here's the current directory tree . ├── bin │ └── /the usual stuff/ ├── include │ └── /the usual stuff/ ├── lib │ └── python2.7 │ └── /the usual stuff/ ├── local │ └── /the usual stuff/ └── README.md 27 directories, 280 files I don't even know where to put the .py files, if I ever make one. What I wanted to do: Make a python module install-able with "pip install ..." If possible, I want a general step by step process on writing Python modules.

Output: A module is a file containing Python definitions and statements. The file name is the module name with the suffix .py create hello.py then write the following function as its content: def helloworld(): print("hello") Then you can import hello: >>> import hello >>> hello.helloworld() 'hello' To group many .py files put them in a folder. Any folder with an __init__.py is considered a module by python and you can call them a package |-HelloModule |_ __init__.py |_ hellomodule.py You can go about with the import statement on your module the usual way. For more information, see 6.4. Packages.


Instruction:
Redirect stdout to a file in Python?

How do I redirect stdout to an arbitrary file in Python? When a long-running Python script (e.g, web application) is started from within the ssh session and backgounded, and the ssh session is closed, the application will raise IOError and fail the moment it tries to write to stdout. I needed to find a way to make the application and modules output to a file rather than stdout to prevent failure due to IOError. Currently, I employ nohup to redirect output to a file, and that gets the job done, but I was wondering if there was a way to do it without using nohup, out of curiosity. I have already tried sys.stdout = open('somefile', 'w'), but this does not seem to prevent some external modules from still outputting to terminal (or maybe the sys.stdout = ... line did not fire at all). I know it should work from simpler scripts I've tested on, but I also didn't have time yet to test on a web application yet.

Output: If you want to do the redirection within the Python script, setting sys.stdout to a file object does the trick: # for python3 import sys with open('file', 'w') as sys.stdout: print('test') A far more common method is to use shell redirection when executing (same on Windows and Linux): $ python3 foo.py > file


Instruction:
Difference in Boto3 between resource, client, and session?

I'm learning how to use AWS SDK for Python (Boto3) from the following resource: https://boto3.readthedocs.io/en/latest/guide/quickstart.html#using-boto-3. My doubt is when to use resource, client, or session, and their respective functionality. I am using Python 2.7.12 in Ubuntu 16.04 LTS.

Output: Client and Resource are two different abstractions within the boto3 SDK for making AWS service requests. If you want to make API calls to an AWS service with boto3, then you do so via a Client or a Resource. You would typically choose to use either the Client abstraction or the Resource abstraction, but you can use both, as needed. I've outlined the differences below to help readers decide which to use. Session is largely orthogonal to the concepts of Client and Resource (but is used by both). Here's some more detailed information on what Client, Resource, and Session are all about. Client this is the original boto3 API abstraction it provides low-level AWS service access all AWS service operations are supported by clients it exposes botocore client to the developer it typically maps 1:1 with the AWS service API it exposes snake-cased method names (e.g. ListBuckets API => list_buckets method) typically yields primitive, non-marshalled data (e.g. DynamoDB attributes are dicts representing primitive DynamoDB values) requires you to code result pagination it is generated from an AWS service description Here's an example of client-level access to an S3 bucket's objects: import boto3 client = boto3.client('s3') response = client.list_objects_v2(Bucket='mybucket') for content in response['Contents']: obj_dict = client.get_object(Bucket='mybucket', Key=content['Key']) print(content['Key'], obj_dict['LastModified']) Note: this client-level code is limited to listing at most 1000 objects. You would have to use a paginator, or implement your own loop, calling list_objects_v2() repeatedly with a continuation marker if there were more than 1000 objects. OK, so that's the low-level Client interface. Now onto the higher-level (more abstract) Resource interface. Resource this is the newer boto3 API abstraction it provides a high-level, object-oriented API it does not provide 100% API coverage of AWS services it uses identifiers and attributes it has actions (operations on resources) it exposes sub-resources and collections of AWS resources typically yields marshalled data, not primitive AWS data (e.g. DynamoDB attributes are native Python values representing primitive DynamoDB values) does result pagination for you it is generated from an AWS resource description Here's the equivalent example using resource-level access to an S3 bucket's objects: import boto3 s3 = boto3.resource('s3') bucket = s3.Bucket('mybucket') for obj in bucket.objects.all(): print(obj.key, obj.last_modified) Note: in this case you do not have to make a second API call to get the objects; they're available to you as a collection on the bucket. These collections of sub-resources are lazily-loaded. You can see that the Resource version of the code is much simpler, more compact, and has more capability (for example it does pagination for you and it exposes properties instead of a raw dictionary). The Client version of the code would actually be more complicated than shown above if you wanted to include pagination. Finally, onto Session which is fundamental to both Client and Resource and how both get access to AWS credentials, for example. Session stores configuration information (primarily credentials and selected region) allows you to create service clients and resources boto3 creates a default session for you when needed A useful resource to learn more about these boto3 concepts is the introductory re:Invent video. Update January 2023 Per the Resources page in the boto3 documentation: The AWS Python SDK team does not intend to add new features to the resources interface in boto3. Existing interfaces will continue to operate during boto3's lifecycle. Customers can find access to newer service features through the client interface. You can read more about the plans to maintain but no longer enhance resources at boto3/discussions/3563.


Instruction:
Creating a Pandas DataFrame from a Numpy array: How do I specify the index column and column headers?

I have a Numpy array consisting of a list of lists, representing a two-dimensional array with row labels and column names as shown below: data = np.array([['','Col1','Col2'],['Row1',1,2],['Row2',3,4]]) I'd like the resulting DataFrame to have Row1 and Row2 as index values, and Col1, Col2 as header values. I can specify the index as follows: df = pd.DataFrame(data, index=data[:,0]) However, I am unsure how to best assign column headers.

Output: Specify data, index and columns to the DataFrame constructor, as follows: >>> pd.DataFrame(data=data[1:,1:], # values ... index=data[1:,0], # 1st column as index ... columns=data[0,1:]) # 1st row as the column names As @joris mentions, you may need to change above to np.int_(data[1:,1:]) to have the correct data type.


Instruction:
Convert NumPy array to Python list

How do I convert a NumPy array into a Python List?

Output: Use tolist(): >>> import numpy as np >>> np.array([[1,2,3],[4,5,6]]).tolist() [[1, 2, 3], [4, 5, 6]] Note that this converts the values from whatever numpy type they may have (e.g. np.int32 or np.float32) to the "nearest compatible Python type" (in a list). If you want to preserve the numpy data types, you could call list() on your array instead, and you'll end up with a list of numpy scalars. (Thanks to Mr_and_Mrs_D for pointing that out in a comment.)


Instruction:
How to convert local time string to UTC?

How do I convert a datetime string in local time to a string in UTC time? I'm sure I've done this before, but can't find it and SO will hopefully help me (and others) do that in future. Clarification: For example, if I have 2008-09-17 14:02:00 in my local timezone (+10), I'd like to generate a string with the equivalent UTC time: 2008-09-17 04:02:00. Also, from http://lucumr.pocoo.org/2011/7/15/eppur-si-muove/, note that in general this isn't possible as with DST and other issues there is no unique conversion from local time to UTC time.

Output: Thanks @rofly, the full conversion from string to string is as follows: import time time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime(time.mktime(time.strptime("2008-09-17 14:04:00", "%Y-%m-%d %H:%M:%S")))) My summary of the time/calendar functions: time.strptime string --> tuple (no timezone applied, so matches string) time.mktime local time tuple --> seconds since epoch (always local time) time.gmtime seconds since epoch --> tuple in UTC and calendar.timegm tuple in UTC --> seconds since epoch time.localtime seconds since epoch --> tuple in local timezone


Instruction:
How can I parse (read) and use JSON in Python?

My Python program receives JSON data, and I need to get bits of information out of it. How can I parse the data and use the result? I think I need to use json.loads for this task, but I can't understand how to do it. For example, suppose that I have jsonStr = '{"one" : "1", "two" : "2", "three" : "3"}'. Given this JSON, and an input of "two", how can I get the corresponding data, "2"? Beware that .load is for files; .loads is for strings. See also: Reading JSON from a file. Occasionally, a JSON document is intended to represent tabular data. If you have something like this and are trying to use it with Pandas, see Python - How to convert JSON File to Dataframe. Some data superficially looks like JSON, but is not JSON. For example, sometimes the data comes from applying repr to native Python data structures. The result may use quotes differently, use title-cased True and False rather than JSON-mandated true and false, etc. For such data, see Convert a String representation of a Dictionary to a dictionary or How to convert string representation of list to a list. Another common variant format puts separate valid JSON-formatted data on each line of the input. (Proper JSON cannot be parsed line by line, because it uses balanced brackets that can be many lines apart.) This format is called JSONL. See Loading JSONL file as JSON objects. Sometimes JSON data from a web source is padded with some extra text. In some contexts, this works around security restrictions in browsers. This is called JSONP and is described at What is JSONP, and why was it created?. In other contexts, the extra text implements a security measure, as described at Why does Google prepend while(1); to their JSON responses?. Either way, handling this in Python is straightforward: simply identify and remove the extra text, and proceed as before.

Output: Very simple: import json data = json.loads('{"one" : "1", "two" : "2", "three" : "3"}') print(data['two']) # or `print data['two']` in Python 2


Instruction:
Assign output of os.system to a variable and prevent it from being displayed on the screen

I want to assign the output of a command I run using os.system to a variable and prevent it from being output to the screen. But, in the below code ,the output is sent to the screen and the value printed for var is 0, which I guess signifies whether the command ran successfully or not. Is there any way to assign the command output to the variable and also stop it from being displayed on the screen? var = os.system("cat /etc/services") print var #Prints 0

Output: From this question which I asked a long time ago, what you may want to use is popen: os.popen('cat /etc/services').read() From the docs for Python 3.6, This is implemented using subprocess.Popen; see that class’s documentation for more powerful ways to manage and communicate with subprocesses. Here's the corresponding code for subprocess: import subprocess proc = subprocess.Popen(["cat", "/etc/services"], stdout=subprocess.PIPE, shell=True) (out, err) = proc.communicate() print("program output:", out)


Instruction:
Converting Epoch time into the datetime

I am getting a response from the rest is an Epoch time format like start_time = 1234566 end_time = 1234578 I want to convert that epoch seconds in MySQL format time so that I could store the differences in my MySQL database. I tried: >>> import time >>> time.gmtime(123456) time.struct_time(tm_year=1970, tm_mon=1, tm_mday=2, tm_hour=10, tm_min=17, tm_sec=36, tm_wday=4, tm_yday=2, tm_isdst=0) The above result is not what I am expecting. I want it be like 2012-09-12 21:00:00 Please suggest how can I achieve this? Also, Why I am getting TypeError: a float is required for >>> getbbb_class.end_time = 1347516459425 >>> mend = time.gmtime(getbbb_class.end_time).tm_hour Traceback (most recent call last): ... TypeError: a float is required

Output: To convert your time value (float or int) to a formatted string, use: strftime('%Y-%m-%d %H:%M:%S', localtime(1347517370)) preceded by this import: from time import strftime, localtime


Instruction:
Why does range(start, end) not include end?

>>> range(1,11) gives you [1,2,3,4,5,6,7,8,9,10] Why not 1-11? Did they just decide to do it like that at random or does it have some value I am not seeing?

Output: Because it's more common to call range(0, 10) which returns [0,1,2,3,4,5,6,7,8,9] which contains 10 elements which equals len(range(0, 10)). There's a tendency in programming to use 0-based indexing. Also, consider the following common code snippet: for i in range(len(li)): pass Could you see that if range() went up to exactly len(li) that this would be problematic? The programmer would need to explicitly subtract 1. This also follows the common trend of programmers preferring for(int i = 0; i < 10; i++) over for(int i = 0; i <= 9; i++). If you are calling range with a start of 1 frequently, you might want to define your own function: >>> def range1(start, end): ... return range(start, end+1) ... >>> range1(1, 10) [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]


Instruction:
Auto reloading python Flask app upon code changes

I'm investigating how to develop a decent web app with Python. Since I don't want some high-order structures to get in my way, my choice fell on the lightweight Flask framework. Time will tell if this was the right choice. So, now I've set up an Apache server with mod_wsgi, and my test site is running fine. However, I'd like to speed up the development routine by making the site automatically reload upon any changes in py or template files I make. I see that any changes in site's .wsgi file causes reloading (even without WSGIScriptReloading On in the apache config file), but I still have to prod it manually (ie, insert extra linebreak, save). Is there some way how to cause reload when I edit some of the app's py files? Or, I am expected to use IDE that refreshes the .wsgi file for me?

Output: Run the flask run CLI command with debug mode enabled, which will automatically enable the reloader. As of Flask 2.2, you can pass --app and --debug options on the command line. $ flask --app main.py --debug run --app can also be set to module:app or module:create_app instead of module.py. See the docs for a full explanation. More options are available with: $ flask run --help Prior to Flask 2.2, you needed to set the FLASK_APP and FLASK_ENV=development environment variables. $ export FLASK_APP=main.py $ export FLASK_ENV=development $ flask run It is still possible to set FLASK_APP and FLASK_DEBUG=1 in Flask 2.2.


Instruction:
Pandas DataFrame to List of Dictionaries

I have the following DataFrame: customer item1 item2 item3 1 apple milk tomato 2 water orange potato 3 juice mango chips which I want to translate it to list of dictionaries per row rows = [ { 'customer': 1, 'item1': 'apple', 'item2': 'milk', 'item3': 'tomato' }, { 'customer': 2, 'item1': 'water', 'item2': 'orange', 'item3': 'potato' }, { 'customer': 3, 'item1': 'juice', 'item2': 'mango', 'item3': 'chips' } ]

Output: Use df.to_dict('records') -- gives the output without having to transpose externally. In [2]: df.to_dict('records') Out[2]: [{'customer': 1L, 'item1': 'apple', 'item2': 'milk', 'item3': 'tomato'}, {'customer': 2L, 'item1': 'water', 'item2': 'orange', 'item3': 'potato'}, {'customer': 3L, 'item1': 'juice', 'item2': 'mango', 'item3': 'chips'}]


Instruction:
Add list to set

How do I add a list of values to an existing set?

Output: You can't add a list to a set because lists are mutable, meaning that you can change the contents of the list after adding it to the set. You can however add tuples to the set, because you cannot change the contents of a tuple: >>> a.add(('f', 'g')) >>> print a set(['a', 'c', 'b', 'e', 'd', ('f', 'g')]) Edit: some explanation: The documentation defines a set as an unordered collection of distinct hashable objects. The objects have to be hashable so that finding, adding and removing elements can be done faster than looking at each individual element every time you perform these operations. The specific algorithms used are explained in the Wikipedia article. Pythons hashing algorithms are explained on effbot.org and pythons __hash__ function in the python reference. Some facts: Set elements as well as dictionary keys have to be hashable Some unhashable datatypes: list: use tuple instead set: use frozenset instead dict: has no official counterpart, but there are some recipes Object instances are hashable by default with each instance having a unique hash. You can override this behavior as explained in the python reference.


Instruction:
What does axis in pandas mean?

Here is my code to generate a dataframe: import pandas as pd import numpy as np dff = pd.DataFrame(np.random.randn(1, 2), columns=list('AB')) then I got the dataframe: A B 0 0.626386 1.52325 When I type the command dff.mean(axis=1), I get: 0 1.074821 dtype: float64 According to the reference of pandas, axis=1 stands for columns and I expect the result of the command to be A 0.626386 B 1.523255 dtype: float64 So what does axis in pandas mean?

Output: It specifies the axis along which the means are computed. By default axis=0. This is consistent with the numpy.mean usage when axis is specified explicitly (in numpy.mean, axis==None by default, which computes the mean value over the flattened array) , in which axis=0 along the rows (namely, index in pandas), and axis=1 along the columns. For added clarity, one may choose to specify axis='index' (instead of axis=0) or axis='columns' (instead of axis=1). A B 0 0.626386 1.52325 → → axis=1 → → ↓ ↓ ↓ axis=0 ↓ ↓ ↓


Instruction:
How can I percent-encode URL parameters in Python?

If I do url = "http://example.com?p=" + urllib.quote(query) It doesn't encode / to %2F (breaks OAuth normalization) It doesn't handle Unicode (it throws an exception) Is there a better library?

Output: From the Python 3 documentation: urllib.parse.quote(string, safe='/', encoding=None, errors=None) Replace special characters in string using the %xx escape. Letters, digits, and the characters '_.-~' are never quoted. By default, this function is intended for quoting the path section of a URL. The optional safe parameter specifies additional ASCII characters that should not be quoted — its default value is '/'. That means passing '' for safe will solve your first issue: >>> import urllib.parse >>> urllib.parse.quote('/test') '/test' >>> urllib.parse.quote('/test', safe='') '%2Ftest' (The function quote was moved from urllib to urllib.parse in Python 3.) By the way, have a look at urlencode. About the second issue, there was a bug report about it and it was fixed in Python 3. For Python 2, you can work around it by encoding as UTF-8 like this: >>> query = urllib.quote(u"Müller".encode('utf8')) >>> print urllib.unquote(query).decode('utf8') Müller


Instruction:
Unicode (UTF-8) reading and writing to files in Python

I'm having some brain failure in understanding reading and writing text to a file (Python 2.4). # The string, which has an a-acute in it. ss = u'Capit\xe1n' ss8 = ss.encode('utf8') repr(ss), repr(ss8) ("u'Capit\xe1n'", "'Capit\xc3\xa1n'") print ss, ss8 print >> open('f1','w'), ss8 >>> file('f1').read() 'Capit\xc3\xa1n\n' So I type in Capit\xc3\xa1n into my favorite editor, in file f2. Then: >>> open('f1').read() 'Capit\xc3\xa1n\n' >>> open('f2').read() 'Capit\\xc3\\xa1n\n' >>> open('f1').read().decode('utf8') u'Capit\xe1n\n' >>> open('f2').read().decode('utf8') u'Capit\\xc3\\xa1n\n' What am I not understanding here? Clearly there is some vital bit of magic (or good sense) that I'm missing. What does one type into text files to get proper conversions? What I'm truly failing to grok here, is what the point of the UTF-8 representation is, if you can't actually get Python to recognize it, when it comes from outside. Maybe I should just JSON dump the string, and use that instead, since that has an asciiable representation! More to the point, is there an ASCII representation of this Unicode object that Python will recognize and decode, when coming in from a file? If so, how do I get it? >>> print simplejson.dumps(ss) '"Capit\u00e1n"' >>> print >> file('f3','w'), simplejson.dumps(ss) >>> simplejson.load(open('f3')) u'Capit\xe1n'

Output: In the notation u'Capit\xe1n\n' (should be just 'Capit\xe1n\n' in 3.x, and must be in 3.0 and 3.1), the \xe1 represents just one character. \x is an escape sequence, indicating that e1 is in hexadecimal. Writing Capit\xc3\xa1n into the file in a text editor means that it actually contains \xc3\xa1. Those are 8 bytes and the code reads them all. We can see this by displaying the result: # Python 3.x - reading the file as bytes rather than text, # to ensure we see the raw data >>> open('f2', 'rb').read() b'Capit\\xc3\\xa1n\n' # Python 2.x >>> open('f2').read() 'Capit\\xc3\\xa1n\n' Instead, just input characters like á in the editor, which should then handle the conversion to UTF-8 and save it. In 2.x, a string that actually contains these backslash-escape sequences can be decoded using the string_escape codec: # Python 2.x >>> print 'Capit\\xc3\\xa1n\n'.decode('string_escape') Capitán The result is a str that is encoded in UTF-8 where the accented character is represented by the two bytes that were written \\xc3\\xa1 in the original string. To get a unicode result, decode again with UTF-8. In 3.x, the string_escape codec is replaced with unicode_escape, and it is strictly enforced that we can only encode from a str to bytes, and decode from bytes to str. unicode_escape needs to start with a bytes in order to process the escape sequences (the other way around, it adds them); and then it will treat the resulting \xc3 and \xa1 as character escapes rather than byte escapes. As a result, we have to do a bit more work: # Python 3.x >>> 'Capit\\xc3\\xa1n\n'.encode('ascii').decode('unicode_escape').encode('latin-1').decode('utf-8') 'Capitán\n'


Instruction:
Apply multiple functions to multiple groupby columns

The docs show how to apply multiple functions on a groupby object at a time using a dict with the output column names as the keys: In [563]: grouped['D'].agg({'result1' : np.sum, .....: 'result2' : np.mean}) .....: Out[563]: result2 result1 A bar -0.579846 -1.739537 foo -0.280588 -1.402938 However, this only works on a Series groupby object. And when a dict is similarly passed to a groupby DataFrame, it expects the keys to be the column names that the function will be applied to. What I want to do is apply multiple functions to several columns (but certain columns will be operated on multiple times). Also, some functions will depend on other columns in the groupby object (like sumif functions). My current solution is to go column by column, and doing something like the code above, using lambdas for functions that depend on other rows. But this is taking a long time, (I think it takes a long time to iterate through a groupby object). I'll have to change it so that I iterate through the whole groupby object in a single run, but I'm wondering if there's a built in way in pandas to do this somewhat cleanly. For example, I've tried something like grouped.agg({'C_sum' : lambda x: x['C'].sum(), 'C_std': lambda x: x['C'].std(), 'D_sum' : lambda x: x['D'].sum()}, 'D_sumifC3': lambda x: x['D'][x['C'] == 3].sum(), ...) but as expected I get a KeyError (since the keys have to be a column if agg is called from a DataFrame). Is there any built in way to do what I'd like to do, or a possibility that this functionality may be added, or will I just need to iterate through the groupby manually?

Output: The second half of the currently accepted answer is outdated and has two deprecations. First and most important, you can no longer pass a dictionary of dictionaries to the agg groupby method. Second, never use .ix. If you desire to work with two separate columns at the same time I would suggest using the apply method which implicitly passes a DataFrame to the applied function. Let's use a similar dataframe as the one from above df = pd.DataFrame(np.random.rand(4,4), columns=list('abcd')) df['group'] = [0, 0, 1, 1] df a b c d group 0 0.418500 0.030955 0.874869 0.145641 0 1 0.446069 0.901153 0.095052 0.487040 0 2 0.843026 0.936169 0.926090 0.041722 1 3 0.635846 0.439175 0.828787 0.714123 1 A dictionary mapped from column names to aggregation functions is still a perfectly good way to perform an aggregation. df.groupby('group').agg({'a':['sum', 'max'], 'b':'mean', 'c':'sum', 'd': lambda x: x.max() - x.min()}) a b c d sum max mean sum <lambda> group 0 0.864569 0.446069 0.466054 0.969921 0.341399 1 1.478872 0.843026 0.687672 1.754877 0.672401 If you don't like that ugly lambda column name, you can use a normal function and supply a custom name to the special __name__ attribute like this: def max_min(x): return x.max() - x.min() max_min.__name__ = 'Max minus Min' df.groupby('group').agg({'a':['sum', 'max'], 'b':'mean', 'c':'sum', 'd': max_min}) a b c d sum max mean sum Max minus Min group 0 0.864569 0.446069 0.466054 0.969921 0.341399 1 1.478872 0.843026 0.687672 1.754877 0.672401 Using apply and returning a Series Now, if you had multiple columns that needed to interact together then you cannot use agg, which implicitly passes a Series to the aggregating function. When using apply the entire group as a DataFrame gets passed into the function. I recommend making a single custom function that returns a Series of all the aggregations. Use the Series index as labels for the new columns: def f(x): d = {} d['a_sum'] = x['a'].sum() d['a_max'] = x['a'].max() d['b_mean'] = x['b'].mean() d['c_d_prodsum'] = (x['c'] * x['d']).sum() return pd.Series(d, index=['a_sum', 'a_max', 'b_mean', 'c_d_prodsum']) df.groupby('group').apply(f) a_sum a_max b_mean c_d_prodsum group 0 0.864569 0.446069 0.466054 0.173711 1 1.478872 0.843026 0.687672 0.630494 If you are in love with MultiIndexes, you can still return a Series with one like this: def f_mi(x): d = [] d.append(x['a'].sum()) d.append(x['a'].max()) d.append(x['b'].mean()) d.append((x['c'] * x['d']).sum()) return pd.Series(d, index=[['a', 'a', 'b', 'c_d'], ['sum', 'max', 'mean', 'prodsum']]) df.groupby('group').apply(f_mi) a b c_d sum max mean prodsum group 0 0.864569 0.446069 0.466054 0.173711 1 1.478872 0.843026 0.687672 0.630494


Instruction:
Simple Digit Recognition OCR in OpenCV-Python

I am trying to implement a "Digit Recognition OCR" in OpenCV-Python (cv2). It is just for learning purposes. I would like to learn both KNearest and SVM features in OpenCV. I have 100 samples (i.e. images) of each digit. I would like to train with them. There is a sample letter_recog.py that comes with OpenCV sample. But I still couldn't figure out on how to use it. I don't understand what are the samples, responses etc. Also, it loads a txt file at first, which I didn't understand first. Later on searching a little bit, I could find a letter_recognition.data in cpp samples. I used it and made a code for cv2.KNearest in the model of letter_recog.py (just for testing): import numpy as np import cv2 fn = 'letter-recognition.data' a = np.loadtxt(fn, np.float32, delimiter=',', converters={ 0 : lambda ch : ord(ch)-ord('A') }) samples, responses = a[:,1:], a[:,0] model = cv2.KNearest() retval = model.train(samples,responses) retval, results, neigh_resp, dists = model.find_nearest(samples, k = 10) print results.ravel() It gave me an array of size 20000, I don't understand what it is. Questions: 1) What is letter_recognition.data file? How to build that file from my own data set? 2) What does results.reval() denote? 3) How we can write a simple digit recognition tool using letter_recognition.data file (either KNearest or SVM)?

Output: Well, I decided to workout myself on my question to solve the above problem. What I wanted is to implement a simple OCR using KNearest or SVM features in OpenCV. And below is what I did and how. (it is just for learning how to use KNearest for simple OCR purposes). 1) My first question was about letter_recognition.data file that comes with OpenCV samples. I wanted to know what is inside that file. It contains a letter, along with 16 features of that letter. And this SOF helped me to find it. These 16 features are explained in the paper Letter Recognition Using Holland-Style Adaptive Classifiers. (Although I didn't understand some of the features at the end) 2) Since I knew, without understanding all those features, it is difficult to do that method. I tried some other papers, but all were a little difficult for a beginner. So I just decided to take all the pixel values as my features. (I was not worried about accuracy or performance, I just wanted it to work, at least with the least accuracy) I took the below image for my training data: (I know the amount of training data is less. But, since all letters are of the same font and size, I decided to try on this). To prepare the data for training, I made a small code in OpenCV. It does the following things: It loads the image. Selects the digits (obviously by contour finding and applying constraints on area and height of letters to avoid false detections). Draws the bounding rectangle around one letter and wait for key press manually. This time we press the digit key ourselves corresponding to the letter in the box. Once the corresponding digit key is pressed, it resizes this box to 10x10 and saves all 100 pixel values in an array (here, samples) and corresponding manually entered digit in another array(here, responses). Then save both the arrays in separate .txt files. At the end of the manual classification of digits, all the digits in the training data (train.png) are labeled manually by ourselves, image will look like below: Below is the code I used for the above purpose (of course, not so clean): import sys import numpy as np import cv2 im = cv2.imread('pitrain.png') im3 = im.copy() gray = cv2.cvtColor(im,cv2.COLOR_BGR2GRAY) blur = cv2.GaussianBlur(gray,(5,5),0) thresh = cv2.adaptiveThreshold(blur,255,1,1,11,2) ################# Now finding Contours ################### contours,hierarchy = cv2.findContours(thresh,cv2.RETR_LIST,cv2.CHAIN_APPROX_SIMPLE) samples = np.empty((0,100)) responses = [] keys = [i for i in range(48,58)] for cnt in contours: if cv2.contourArea(cnt)>50: [x,y,w,h] = cv2.boundingRect(cnt) if h>28: cv2.rectangle(im,(x,y),(x+w,y+h),(0,0,255),2) roi = thresh[y:y+h,x:x+w] roismall = cv2.resize(roi,(10,10)) cv2.imshow('norm',im) key = cv2.waitKey(0) if key == 27: # (escape to quit) sys.exit() elif key in keys: responses.append(int(chr(key))) sample = roismall.reshape((1,100)) samples = np.append(samples,sample,0) responses = np.array(responses,np.float32) responses = responses.reshape((responses.size,1)) print "training complete" np.savetxt('generalsamples.data',samples) np.savetxt('generalresponses.data',responses) Now we enter in to training and testing part. For the testing part, I used the below image, which has the same type of letters I used for the training phase. For training we do as follows: Load the .txt files we already saved earlier create an instance of the classifier we are using (it is KNearest in this case) Then we use KNearest.train function to train the data For testing purposes, we do as follows: We load the image used for testing process the image as earlier and extract each digit using contour methods Draw a bounding box for it, then resize it to 10x10, and store its pixel values in an array as done earlier. Then we use KNearest.find_nearest() function to find the nearest item to the one we gave. ( If lucky, it recognizes the correct digit.) I included last two steps (training and testing) in single code below: import cv2 import numpy as np ####### training part ############### samples = np.loadtxt('generalsamples.data',np.float32) responses = np.loadtxt('generalresponses.data',np.float32) responses = responses.reshape((responses.size,1)) model = cv2.KNearest() model.train(samples,responses) ############################# testing part ######################### im = cv2.imread('pi.png') out = np.zeros(im.shape,np.uint8) gray = cv2.cvtColor(im,cv2.COLOR_BGR2GRAY) thresh = cv2.adaptiveThreshold(gray,255,1,1,11,2) contours,hierarchy = cv2.findContours(thresh,cv2.RETR_LIST,cv2.CHAIN_APPROX_SIMPLE) for cnt in contours: if cv2.contourArea(cnt)>50: [x,y,w,h] = cv2.boundingRect(cnt) if h>28: cv2.rectangle(im,(x,y),(x+w,y+h),(0,255,0),2) roi = thresh[y:y+h,x:x+w] roismall = cv2.resize(roi,(10,10)) roismall = roismall.reshape((1,100)) roismall = np.float32(roismall) retval, results, neigh_resp, dists = model.find_nearest(roismall, k = 1) string = str(int((results[0][0]))) cv2.putText(out,string,(x,y+h),0,1,(0,255,0)) cv2.imshow('im',im) cv2.imshow('out',out) cv2.waitKey(0) And it worked, below is the result I got: Here it worked with 100% accuracy. I assume this is because all the digits are of the same kind and the same size. But anyway, this is a good start to go for beginners (I hope so).


Instruction:
CMD opens Windows Store when I type 'python'

Today when I tried to run simple code on Sublime Text 3, the following message appeared: Python was not found but can be installed from the Microsoft Store: https://go.microsoft.com/fwlink?linkID=2082640 And when I type Python in CMD, it opens the Windows Store for me to download Python 3.7. This problem started today for no good reason. I didn't change or download anything about Python and already tried reinstalling Python, and the Path environment variable is correct.

Output: Use the Windows search bar to find "Manage app execution aliases". There should be two aliases for Python. Unselect them, and this will allow the usual Python aliases "python" and "python3". See the image below. I think we have this problem when installing Python because in a new Windows installation the aliases are in the ON position as in image below. When turned on, Windows puts an empty or fake file named python.exe and python3.exe in the directory named %USERPROFILE%\AppData\Local\Microsoft\WindowsApps. This is the alias. Then Microsoft put that directory at the top of the list in the "Path" environment variables. When you enter "python" in cmd, it searches the directories listed in your "Path" environment variables page from top to bottom. So if you installed Python after a new Windows 10 install then get redirected to the Windows Store, it's because there are two python.exe's: The alias in the App Execution Alias page, and the real one wherever you installed Python. But cmd finds the App execution, alias python.exe, first because that directory is at the top of the Path. I think the easiest solution is to just check the python.exe and python3.exe to OFF as I suggested before, which deletes the fake EXE file files. The first time I ran into this problem, I manually deleted the python.exe and python3.exe files but when I restarted the files regenerated. That prompted me to search for the App Execution Aliases page and uncheck the box, which solved it for me, by not allowing the files to regenerate. Based on this Microsoft Devblog, they stated they created this system partially for new Python users, specifically kids learning Python in school that had trouble installing it, and focus on learning to code. I think Windows probably deletes those aliases if you install Python from the Windows App Store. We are noticing that they do not get deleted if you manually install from another source. (Also, the empty/fake python.exe is not really empty. It says 0 KB in the screenshot, but entering "start ms-windows-store:" in cmd opens the Windows App Store, so it probably just has a line with that and a way to direct it to the Python page.) One alternative, as Chipjust suggested, you can create a new alias for Python using something like DOSKEY as explained in this article for example: How to set aliases for the command prompt in Windows Another alternative is to delete the user path environment variable that points to the alias files, %USERPROFILE%\AppData\Local\Microsoft\WindowsApps, but the App Execution Aliases handle more apps than just python, and deleting the path from environment variables breaks all the other apps that have execution aliases in that directory; which on my PC includes notepad, xbox game bar, spotify, monitoring software for my motherboard, paint, windows subsystem for android, to name a few. Also if you think about it, the average Windows user is unfamiliar editing environment variables and on school and business owned computers requires administrative access. So deleting the path to ...\WindowsApps, from the path environment variable, is not ideal.


Instruction:
What is pyproject.toml file for?

Background I was about to try Python package downloaded from GitHub, and realized that it did not have a setup.py, so I could not install it with pip install -e <folder> Instead, the package had a pyproject.toml file which seems to have very similar entries as the setup.py usually has. What I found Googling lead me into PEP-518 and it gives some critique to setup.py in Rationale section. However, it does not clearly tell that usage of setup.py should be avoided, or that pyproject.toml would as such completely replace setup.py. Questions Is the pyproject.toml something that is used to replace setup.py? Or should a package come with both, a pyproject.toml and a setup.py? How would one install a project with pyproject.toml in an editable state?

Output: What is it for? Currently there are multiple packaging tools being popular in Python community and while setuptools still seems to be prevalent it's not a de facto standard anymore. This situation creates a number of hassles for both end users and developers: For setuptools-based packages installation from source / build of a distribution can fail if one doesn't have setuptools installed; pip doesn't support the installation of packages based on other packaging tools from source, so these tools had to generate a setup.py file to produce a compatible package. To build a distribution package one has to install the packaging tool first and then use tool-specific commands; If package author decides to change the packaging tool, workflows must be changed as well to use different tool-specific commands. pyproject.toml is a new configuration file introduced by PEP 517 and PEP 518 to solve these problems: ... think of the (rough) steps required to produce a built artifact for a project: The source checkout of the project. Installation of the build system. Execute the build system. This PEP [518] covers step #2. PEP 517 covers step #3 ... Any tool can also extend this file with its own section (table) to accept tool-specific options, but it's up to them and not required. PEP 621 suggests using pyproject.toml to specify package core metadata in static, tool-agnostic way. Which backends currently support this is shown in the following table: enscons flit_core hatchling pdm-backend poetry-core setuptools 0.26.0+ 3.2+ 0.3+ 0.3.0+ 2.0.0+ 61.0.0+ Does it replace setup.py? For setuptools-based packages pyproject.toml is not strictly meant to replace setup.py, but rather to ensure its correct execution if it's still needed. For other packaging tools – yes, it is: Where the build-backend key exists, this takes precedence and the source tree follows the format and conventions of the specified backend (as such no setup.py is needed unless the backend requires it). Projects may still wish to include a setup.py for compatibility with tools that do not use this spec. How to install a package in editable mode? Originally "editable install" was a setuptools-specific feature and as such it was not supported by PEP 517. Later on PEP 660 extended this concept to packages using pyproject.toml. There are two possible conditions for installing a package in editable mode using pip: Modern: Both the frontend (pip) and a backend must support PEP 660. pip supports it since version 21.3; Legacy: Packaging tool must provide a setup.py file which supports the develop command. Since version 21.1 pip can also install packages using only setup.cfg file in editable mode. The following table describes the support of editable installs by various backends: enscons flit_core hatchling pdm-backend poetry-core setuptools 0.28.0+ 3.4+ 0.3+ 0.8.0+ 1.0.8+ 64.0.0+


Instruction:
Can "list_display" in a Django ModelAdmin display attributes of ForeignKey fields?

I have a Person model that has a foreign key relationship to Book, which has a number of fields, but I'm most concerned about author (a standard CharField). With that being said, in my PersonAdmin model, I'd like to display book.author using list_display: class PersonAdmin(admin.ModelAdmin): list_display = ['book.author',] I've tried all of the obvious methods for doing so, but nothing seems to work. Any suggestions?

Output: As another option, you can do lookups like: #models.py class UserAdmin(admin.ModelAdmin): list_display = (..., 'get_author') def get_author(self, obj): return obj.book.author get_author.short_description = 'Author' get_author.admin_order_field = 'book__author' For Django 3.2 or higher, please refer to this answer


Instruction:
python: how to identify if a variable is an array or a scalar

I have a function that takes the argument NBins. I want to make a call to this function with a scalar 50 or an array [0, 10, 20, 30]. How can I identify within the function, what the length of NBins is? or said differently, if it is a scalar or a vector? I tried this: >>> N=[2,3,5] >>> P = 5 >>> len(N) 3 >>> len(P) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: object of type 'int' has no len() >>> As you see, I can't apply len to P, since it's not an array.... Is there something like isarray or isscalar in python? thanks

Output: >>> import collections.abc >>> isinstance([0, 10, 20, 30], collections.abc.Sequence) and not isinstance([0, 10, 20, 30], (str, unicode)) True >>> isinstance(50, collections.abc.Sequence) and not isinstance(50, (str, unicode)) False note: isinstance also supports a tuple of classes, check type(x) in (..., ...) should be avoided and is unnecessary. You may also wanna check not isinstance(x, (str, unicode)) As noted by @2080 and also here this won't work for numpy arrays. eg. >>> import collections.abc >>> import numpy as np >>> isinstance((1, 2, 3), collections.abc.Sequence) True >>> isinstance(np.array([1, 2, 3]), collections.abc.Sequence) False In which case you may try the answer from @jpaddison3: >>> hasattr(np.array([1, 2, 3]), "__len__") True >>> hasattr([1, 2, 3], "__len__") True >>> hasattr((1, 2, 3), "__len__") True However as noted here, this is not perfect either, and will incorrectly (at least according to me) classify dictionaries as sequences whereas isinstance with collections.abc.Sequence classifies correctly: >>> hasattr({"a": 1}, "__len__") True >>> from numpy.distutils.misc_util import is_sequence >>> is_sequence({"a": 1}) True >>> isinstance({"a": 1}, collections.abc.Sequence) False You could customise your solution to something like this, add more types to isinstance depending on your needs: >>> isinstance(np.array([1, 2, 3]), (collections.abc.Sequence, np.ndarray)) True >>> isinstance([1, 2, 3], (collections.abc.Sequence, np.ndarray)) True


Instruction:
Reverse colormap in matplotlib

I would like to know how to simply reverse the color order of a given colormap in order to use it with plot_surface.

Output: The standard colormaps also all have reversed versions. They have the same names with _r tacked on to the end. (Documentation here.)


Instruction:
Is it bad to have my virtualenv directory inside my git repository?

I'm thinking about putting the virtualenv for a Django web app I am making inside my git repository for the app. It seems like an easy way to keep deploy's simple and easy. Is there any reason why I shouldn't do this?

Output: I use pip freeze to get the packages I need into a requirements.txt file and add that to my repository. I tried to think of a way of why you would want to store the entire virtualenv, but I could not.


Instruction:
In requirements.txt, what does tilde equals (~=) mean?

In the requirements.txt for a Python library I am using, one of the requirements is specified like: mock-django~=0.6.10 What does ~= mean?

Output: It means it will select the latest version of the package, greater than or equal to 0.6.10, but still in the 0.6.* version, so it won't download 0.7.0 for example. It ensures you will get security fixes but keep backward-compatibility, if the package maintainer respects the semantic versioning (which states that breaking changes should occur only in major versions). Or, as said by PEP 440: For a given release identifier V.N , the compatible release clause is approximately equivalent to the pair of comparison clauses: >= V.N, == V.* Definition in PEP 440 Complete example here in the documentation


Instruction:
Catching an exception while using a Python 'with' statement

I can't figure out how to handle exception for python 'with' statement. If I have a code: with open("a.txt") as f: print f.readlines() I really want to handle 'file not found exception' in order to do something. But I can't write with open("a.txt") as f: print f.readlines() except: print 'oops' and can't write with open("a.txt") as f: print f.readlines() else: print 'oops' Enclosing with in a try/except statement doesn't work either, and an exception is not raised. What can I do in order to process failure inside with statement in a Pythonic way?

Output: This solution will keep the with-block-code outside of the try-except-clause. try: f = open('foo.txt') except FileNotFoundError: print('error') else: with f: print f.readlines()


Instruction:
How can I mock requests and the response?

I am trying to use Pythons mock package to mock Pythons requests module. What are the basic calls to get me working in below scenario? In my views.py, I have a function that makes variety of requests.get() calls with different response each time def myview(request): res1 = requests.get('aurl') res2 = request.get('burl') res3 = request.get('curl') In my test class I want to do something like this but cannot figure out exact method calls Step 1: # Mock the requests module # when mockedRequests.get('aurl') is called then return 'a response' # when mockedRequests.get('burl') is called then return 'b response' # when mockedRequests.get('curl') is called then return 'c response' Step 2: Call my view Step 3: verify response contains 'a response', 'b response' , 'c response' How can I complete Step 1 (mocking the requests module)?

Output: Here is what worked for me: from unittest import mock @mock.patch('requests.get', mock.Mock(side_effect = lambda k:{'aurl': 'a response', 'burl' : 'b response'}.get(k, 'unhandled request %s'%k)))


Instruction:
Matplotlib different size subplots

I need to add two subplots to a figure. One subplot needs to be about three times as wide as the second (same height). I accomplished this using GridSpec and the colspan argument but I would like to do this using figure so I can save to PDF. I can adjust the first figure using the figsize argument in the constructor, but how do I change the size of the second plot?

Output: As of matplotlib 3.6.0, width_ratios and height_ratios can now be passed directly as keyword arguments to plt.subplots and subplot_mosaic, as per What's new in Matplotlib 3.6.0 (Sep 15, 2022). f, (a0, a1) = plt.subplots(1, 2, width_ratios=[3, 1]) f, (a0, a1, a2) = plt.subplots(3, 1, height_ratios=[1, 1, 3]) Another way is to use the subplots function and pass the width ratio with gridspec_kw matplotlib Tutorial: Customizing Figure Layouts Using GridSpec and Other Functions matplotlib.gridspec.GridSpec has available gridspect_kw options import numpy as np import matplotlib.pyplot as plt # generate some data x = np.arange(0, 10, 0.2) y = np.sin(x) # plot it f, (a0, a1) = plt.subplots(1, 2, gridspec_kw={'width_ratios': [3, 1]}) a0.plot(x, y) a1.plot(y, x) f.tight_layout() f.savefig('grid_figure.pdf') Because the question is canonical, here is an example with vertical subplots. # plot it f, (a0, a1, a2) = plt.subplots(3, 1, gridspec_kw={'height_ratios': [1, 1, 3]}) a0.plot(x, y) a1.plot(x, y) a2.plot(x, y) f.tight_layout()


Instruction:
How can I see the entire HTTP request that's being sent by my Python application?

In my case, I'm using the Requests library to call PayPal's API over HTTPS. Unfortunately, I'm getting an error from PayPal, and PayPal support cannot figure out what the error is or what's causing it. They want me to "Please provide the entire request, headers included". How can I do that?

Output: A simple method: enable logging in recent versions of Requests (1.x and higher.) Requests uses the http.client and logging module configuration to control logging verbosity, as described here. Demonstration Code excerpted from the linked documentation: import requests import logging # These two lines enable debugging at httplib level (requests->urllib3->http.client) # You will see the REQUEST, including HEADERS and DATA, and RESPONSE with HEADERS but without DATA. # The only thing missing will be the response.body which is not logged. try: import http.client as http_client except ImportError: # Python 2 import httplib as http_client http_client.HTTPConnection.debuglevel = 1 # You must initialize logging, otherwise you'll not see debug output. logging.basicConfig() logging.getLogger().setLevel(logging.DEBUG) requests_log = logging.getLogger("requests.packages.urllib3") requests_log.setLevel(logging.DEBUG) requests_log.propagate = True requests.get('https://httpbin.org/headers') Example Output python requests-logging.py Output: INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): httpbin.org send: 'GET /headers HTTP/1.1\r\nHost: httpbin.org\r\nAccept-Encoding: gzip, deflate, compress\r\nAccept: */*\r\nUser-Agent: python-requests/1.2.0 CPython/2.7.3 Linux/3.2.0-48-generic\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: Content-Type: application/json header: Date: Sat, 29 Jun 2013 11:19:34 GMT header: Server: gunicorn/0.17.4 header: Content-Length: 226 header: Connection: keep-alive DEBUG:requests.packages.urllib3.connectionpool:"GET /headers HTTP/1.1" 200 226


Instruction:
How do I specify new lines in a string in order to write multiple lines to a file?

How can I indicate a newline in a string in Python, so that I can write multiple lines to a text file?

Output: It depends on how correct you want to be. \n will usually do the job. If you really want to get it right, you look up the newline character in the os package. (It's actually called linesep.) Note: when writing to files using the Python API, do not use the os.linesep. Just use \n; Python automatically translates that to the proper newline character for your platform.


Instruction:
how do I insert a column at a specific column index in pandas?

Can I insert a column at a specific column index in pandas? import pandas as pd df = pd.DataFrame({'l':['a','b','c','d'], 'v':[1,2,1,2]}) df['n'] = 0 This will put column n as the last column of df, but isn't there a way to tell df to put n at the beginning?

Output: see docs: http://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.insert.html using loc = 0 will insert at the beginning df.insert(loc, column, value) df = pd.DataFrame({'B': [1, 2, 3], 'C': [4, 5, 6]}) df Out: B C 0 1 4 1 2 5 2 3 6 idx = 0 new_col = [7, 8, 9] # can be a list, a Series, an array or a scalar df.insert(loc=idx, column='A', value=new_col) df Out: A B C 0 7 1 4 1 8 2 5 2 9 3 6


Instruction:
Concatenating two one-dimensional NumPy arrays

How do I concatenate two one-dimensional arrays in NumPy? I tried numpy.concatenate: import numpy as np a = np.array([1, 2, 3]) b = np.array([4, 5]) np.concatenate(a, b) But I get an error: TypeError: only length-1 arrays can be converted to Python scalars

Output: Use: np.concatenate([a, b]) The arrays you want to concatenate need to be passed in as a sequence, not as separate arguments. From the NumPy documentation: numpy.concatenate((a1, a2, ...), axis=0) Join a sequence of arrays together. It was trying to interpret your b as the axis parameter, which is why it complained it couldn't convert it into a scalar.


Instruction:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Let x be a NumPy array. The following: (x > 1) and (x < 3) Gives the error message: ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() How do I fix this?

Output: If a and b are Boolean NumPy arrays, the & operation returns the elementwise-and of them: a & b That returns a Boolean array. To reduce this to a single Boolean value, use either (a & b).any() or (a & b).all() Note: if a and b are non-Boolean arrays, consider (a - b).any() or (a - b).all() instead. Rationale The NumPy developers felt there was no one commonly understood way to evaluate an array in Boolean context: it could mean True if any element is True, or it could mean True if all elements are True, or True if the array has non-zero length, just to name three possibilities. Since different users might have different needs and different assumptions, the NumPy developers refused to guess and instead decided to raise a ValueError whenever one tries to evaluate an array in Boolean context. Applying and to two numpy arrays causes the two arrays to be evaluated in Boolean context (by calling __bool__ in Python3 or __nonzero__ in Python2).


Instruction:
Standard way to embed version into Python package?

Is there a standard way to associate version string with a Python package in such way that I could do the following? import foo print(foo.version) I would imagine there's some way to retrieve that data without any extra hardcoding, since minor/major strings are specified in setup.py already. Alternative solution that I found was to have import __version__ in my foo/__init__.py and then have __version__.py generated by setup.py.

Output: Not directly an answer to your question, but you should consider naming it __version__, not version. This is almost a quasi-standard. Many modules in the standard library use __version__, and this is also used in lots of 3rd-party modules, so it's the quasi-standard. Usually, __version__ is a string, but sometimes it's also a float or tuple. As mentioned by S.Lott (Thank you!), PEP 8 says it explicitly: Module Level Dunder Names Module level "dunders" (i.e. names with two leading and two trailing underscores) such as __all__, __author__, __version__, etc. should be placed after the module docstring but before any import statements except from __future__ imports. You should also make sure that the version number conforms to the format described in PEP 440 (PEP 386 a previous version of this standard).


Instruction:
How do I get the object if it exists, or None if it does not exist in Django?

When I ask the model manager to get an object, it raises DoesNotExist when there is no matching object. go = Content.objects.get(name="baby") Instead of DoesNotExist, how can I have go be None instead?

Output: There is no 'built in' way to do this as of version 1.2. Django will raise the DoesNotExist exception every time. The idiomatic way to handle this in Python is to wrap it in a try catch: try: go = SomeModel.objects.get(foo='bar') except SomeModel.DoesNotExist: go = None What I did was to subclass models.Manager, create a safe_get like the code above, and use that manager for my models. That way you can write one line to achieve this: SomeModel.objects.safe_get(foo='bar')


Instruction:
Pythonic way to combine for-loop and if-statement

I know how to use both for loops and if statements on separate lines, such as: >>> a = [2,3,4,5,6,7,8,9,0] ... xyz = [0,12,4,6,242,7,9] ... for x in xyz: ... if x in a: ... print(x) 0,4,6,7,9 And I know I can use a list comprehension to combine these when the statements are simple, such as: print([x for x in xyz if x in a]) But what I can't find is a good example anywhere (to copy and learn from) demonstrating a complex set of commands (not just "print x") that occur following a combination of a for loop and some if statements. Something that I would expect looks like: for x in xyz if x not in a: print(x...) Is this just not the way python is supposed to work?

Output: You can use generator expressions like this: gen = (x for x in xyz if x not in a) for x in gen: print(x)


Instruction:
Find string between two substrings

How do I find a string between two substrings ('123STRINGabc' -> 'STRING')? My current method is like this: >>> start = 'asdf=5;' >>> end = '123jasd' >>> s = 'asdf=5;iwantthis123jasd' >>> print((s.split(start))[1].split(end)[0]) iwantthis However, this seems very inefficient and un-pythonic. What is a better way to do something like this? Forgot to mention: The string might not start and end with start and end. They may have more characters before and after.

Output: import re s = 'asdf=5;iwantthis123jasd' result = re.search('asdf=5;(.*)123jasd', s) print(result.group(1)) # returns 'iwantthis'


Instruction:
What is the purpose of the -m switch?

Could you explain to me what the difference is between calling python -m mymod1 mymod2.py args and python mymod1.py mymod2.py args It seems in both cases mymod1.py is called and sys.argv is ['mymod1.py', 'mymod2.py', 'args'] So what is the -m switch for?

Output: The first line of the Rationale section of PEP 338 says: Python 2.4 adds the command line switch -m to allow modules to be located using the Python module namespace for execution as scripts. The motivating examples were standard library modules such as pdb and profile, and the Python 2.4 implementation is fine for this limited purpose. So you can specify any module in Python's search path this way, not just files in the current directory. You're correct that python mymod1.py mymod2.py args has exactly the same effect. The first line of the Scope of this proposal section states: In Python 2.4, a module located using -m is executed just as if its filename had been provided on the command line. With -m more is possible, like working with modules which are part of a package, etc. That's what the rest of PEP 338 is about. Read it for more info.


Instruction:
Anaconda vs. miniconda

In the Anaconda repository, there are two types of installers: "Anaconda installers" and "Miniconda installers". What are their differences? Besides, for an installer file, Anaconda2-4.4.0.1-Linux-ppc64le.sh, what does 2-4.4.0.1 stand for?

Output: The difference is that miniconda is just shipping the repository management system. So when you install it there is just the management system without packages. Whereas with Anaconda, it is like a distribution with some built in packages. Like with any Linux distribution, there are some releases which bundles lots of updates for the included packages. That is why there is a difference in version numbering. If you only decide to upgrade Anaconda, you are updating a whole system. EDIT there are new options now for on the package management side. mamba can be used as a drop in replacement for conda. It has a faster solver and is a complete re-write in C++. The solver is actually experimentally available in conda with --experimental-solver=libmamba. Keywords to look for if you want to learn more: mamba, mambaforge, micromamba.


Instruction:
How to get only the last part of a path in Python?

In python, suppose I have a path like this: /folderA/folderB/folderC/folderD/ How can I get just the folderD part?

Output: Use os.path.normpath to strip off any trailing slashes, then os.path.basename gives you the last part of the path: >>> os.path.basename(os.path.normpath('/folderA/folderB/folderC/folderD/')) 'folderD' Using only basename gives everything after the last slash, which in this case is ''.


Instruction:
'too many values to unpack', iterating over a dict. key=>string, value=>list

I am getting the too many values to unpack error. Any idea how I can fix this? first_names = ['foo', 'bar'] last_names = ['gravy', 'snowman'] fields = { 'first_names': first_names, 'last_name': last_names, } for field, possible_values in fields: # error happens on this line

Output: Python 3 Use items(). for field, possible_values in fields.items(): print(field, possible_values) Python 2 Use iteritems(). for field, possible_values in fields.iteritems(): print field, possible_values See this answer for more information on iterating through dictionaries, such as using items(), across Python versions. For reference, iteritems() was removed in Python 3.


Instruction:
What is and how can I to use getattr() in Python?

I read an article about the getattr function, but I still can't understand what it's for. The only thing I understand about getattr() is that getattr(li, "pop") is the same as calling li.pop. When and how do I use this exactly? The book said something about using it to get a reference to a function whose name isn't known until runtime, but when and why would I use this?

Output: getattr(object, 'x') is completely equivalent to object.x. There are only two cases where getattr can be useful. you can't write object.x, because you don't know in advance which attribute you want (it comes from a string). It is very useful for metaprogramming. you want to provide a default value. object.y will raise an AttributeError if there's no y. But getattr(object, 'y', 5) will return 5.


Instruction:
How can I partition (split up, divide) a list based on a condition?

I have some code like: good = [x for x in mylist if x in goodvals] bad = [x for x in mylist if x not in goodvals] The goal is to split up the contents of mylist into two other lists, based on whether or not they meet a condition. How can I do this more elegantly? Can I avoid doing two separate iterations over mylist? Can I improve performance by doing so?

Output: good = [x for x in mylist if x in goodvals] bad = [x for x in mylist if x not in goodvals] is there a more elegant way to do this? That code is perfectly readable, and extremely clear! # files looks like: [ ('file1.jpg', 33L, '.jpg'), ('file2.avi', 999L, '.avi'), ... ] IMAGE_TYPES = ('.jpg','.jpeg','.gif','.bmp','.png') images = [f for f in files if f[2].lower() in IMAGE_TYPES] anims = [f for f in files if f[2].lower() not in IMAGE_TYPES] Again, this is fine! There might be slight performance improvements using sets, but it's a trivial difference, and I find the list comprehension far easier to read, and you don't have to worry about the order being messed up, duplicates being removed as so on. In fact, I may go another step "backward", and just use a simple for loop: images, anims = [], [] for f in files: if f.lower() in IMAGE_TYPES: images.append(f) else: anims.append(f) The a list-comprehension or using set() is fine until you need to add some other check or another bit of logic - say you want to remove all 0-byte jpeg's, you just add something like.. if f[1] == 0: continue


Instruction:
Getting list of parameter names inside python function

Is there an easy way to be inside a python function and get a list of the parameter names? For example: def func(a,b,c): print magic_that_does_what_I_want() >>> func() ['a','b','c'] Thanks

Output: Well we don't actually need inspect here. >>> func = lambda x, y: (x, y) >>> >>> func.__code__.co_argcount 2 >>> func.__code__.co_varnames ('x', 'y') >>> >>> def func2(x,y=3): ... print(func2.__code__.co_varnames) ... pass # Other things ... >>> func2(3,3) ('x', 'y') >>> >>> func2.__defaults__ (3,)


Instruction:
Select DataFrame rows between two dates

I am creating a DataFrame from a csv as follows: stock = pd.read_csv('data_in/' + filename + '.csv', skipinitialspace=True) The DataFrame has a date column. Is there a way to create a new DataFrame (or just overwrite the existing one) which only contains rows with date values that fall within a specified date range or between two specified date values?

Output: There are two possible solutions: Use a boolean mask, then use df.loc[mask] Set the date column as a DatetimeIndex, then use df[start_date : end_date] Using a boolean mask: Ensure df['date'] is a Series with dtype datetime64[ns]: df['date'] = pd.to_datetime(df['date']) Make a boolean mask. start_date and end_date can be datetime.datetimes, np.datetime64s, pd.Timestamps, or even datetime strings: #greater than the start date and smaller than the end date mask = (df['date'] > start_date) & (df['date'] <= end_date) Select the sub-DataFrame: df.loc[mask] or re-assign to df df = df.loc[mask] For example, import numpy as np import pandas as pd df = pd.DataFrame(np.random.random((200,3))) df['date'] = pd.date_range('2000-1-1', periods=200, freq='D') mask = (df['date'] > '2000-6-1') & (df['date'] <= '2000-6-10') print(df.loc[mask]) yields 0 1 2 date 153 0.208875 0.727656 0.037787 2000-06-02 154 0.750800 0.776498 0.237716 2000-06-03 155 0.812008 0.127338 0.397240 2000-06-04 156 0.639937 0.207359 0.533527 2000-06-05 157 0.416998 0.845658 0.872826 2000-06-06 158 0.440069 0.338690 0.847545 2000-06-07 159 0.202354 0.624833 0.740254 2000-06-08 160 0.465746 0.080888 0.155452 2000-06-09 161 0.858232 0.190321 0.432574 2000-06-10 Using a DatetimeIndex: If you are going to do a lot of selections by date, it may be quicker to set the date column as the index first. Then you can select rows by date using df.loc[start_date:end_date]. import numpy as np import pandas as pd df = pd.DataFrame(np.random.random((200,3))) df['date'] = pd.date_range('2000-1-1', periods=200, freq='D') df = df.set_index(['date']) print(df.loc['2000-6-1':'2000-6-10']) yields 0 1 2 date 2000-06-01 0.040457 0.326594 0.492136 # <- includes start_date 2000-06-02 0.279323 0.877446 0.464523 2000-06-03 0.328068 0.837669 0.608559 2000-06-04 0.107959 0.678297 0.517435 2000-06-05 0.131555 0.418380 0.025725 2000-06-06 0.999961 0.619517 0.206108 2000-06-07 0.129270 0.024533 0.154769 2000-06-08 0.441010 0.741781 0.470402 2000-06-09 0.682101 0.375660 0.009916 2000-06-10 0.754488 0.352293 0.339337 While Python list indexing, e.g. seq[start:end] includes start but not end, in contrast, Pandas df.loc[start_date : end_date] includes both end-points in the result if they are in the index. Neither start_date nor end_date has to be in the index however. Also note that pd.read_csv has a parse_dates parameter which you could use to parse the date column as datetime64s. Thus, if you use parse_dates, you would not need to use df['date'] = pd.to_datetime(df['date']).


Instruction:
Understanding __get__ and __set__ and Python descriptors

I am trying to understand what Python's descriptors are and what they are useful for. I understand how they work, but here are my doubts. Consider the following code: class Celsius(object): def __init__(self, value=0.0): self.value = float(value) def __get__(self, instance, owner): return self.value def __set__(self, instance, value): self.value = float(value) class Temperature(object): celsius = Celsius() Why do I need the descriptor class? What is instance and owner here? (in __get__). What is the purpose of these parameters? How would I call/use this example?

Output: The descriptor is how Python's property type is implemented. A descriptor simply implements __get__, __set__, etc. and is then added to another class in its definition (as you did above with the Temperature class). For example: temp=Temperature() temp.celsius #calls celsius.__get__ Accessing the property you assigned the descriptor to (celsius in the above example) calls the appropriate descriptor method. instance in __get__ is the instance of the class (so above, __get__ would receive temp, while owner is the class with the descriptor (so it would be Temperature). You need to use a descriptor class to encapsulate the logic that powers it. That way, if the descriptor is used to cache some expensive operation (for example), it could store the value on itself and not its class. The official Python documentation includes an article about descriptors that walks through how they work in more detail, including several examples. EDIT: As jchl pointed out in the comments, if you simply try Temperature.celsius, instance will be None.


Instruction:
How do I get user IP address in Django?

How do I get user's IP in Django? I have a view like this: # Create your views from django.contrib.gis.utils import GeoIP from django.template import RequestContext from django.shortcuts import render_to_response def home(request): g = GeoIP() client_ip = request.META['REMOTE_ADDR'] lat,long = g.lat_lon(client_ip) return render_to_response('home_page_tmp.html',locals()) But I get this error: KeyError at /mypage/ 'REMOTE_ADDR' Request Method: GET Request URL: http://mywebsite.example/mypage/ Django Version: 1.2.4 Exception Type: KeyError Exception Value: 'REMOTE_ADDR' Exception Location: /mysite/homepage/views.py in home, line 9 Python Executable: /usr/bin/python Python Version: 2.6.6 Python Path: ['/mysite', '/usr/local/lib/python2.6/dist-packages/flup-1.0.2-py2.6.egg', '/usr/lib/python2.6', '/usr/lib/python2.6/plat-linux2', '/usr/lib/python2.6/lib-tk', '/usr/lib/python2.6/lib-old', '/usr/lib/python2.6/lib-dynload', '/usr/local/lib/python2.6/dist-packages', '/usr/lib/python2.6/dist-packages', '/usr/lib/pymodules/python2.6'] Server time: Sun, 2 Jan 2011 20:42:50 -0600

Output: def get_client_ip(request): x_forwarded_for = request.META.get('HTTP_X_FORWARDED_FOR') if x_forwarded_for: ip = x_forwarded_for.split(',')[0] else: ip = request.META.get('REMOTE_ADDR') return ip Make sure you have reverse proxy (if any) configured correctly (e.g. mod_rpaf installed for Apache). Note: the above uses the first item in X-Forwarded-For, but you might want to use the last item (e.g., in the case of Heroku: Get client's real IP address on Heroku) And then just pass the request as argument to it; get_client_ip(request) Django documentation for HttpRequest.META


Instruction:
Take the content of a list and append it to another list

I am trying to understand if it makes sense to take the content of a list and append it to another list. I have the first list created through a loop function, that will get specific lines out of a file and will save them in a list. Then a second list is used to save these lines, and start a new cycle over another file. My idea was to get the list once that the for cycle is done, dump it into the second list, then start a new cycle, dump the content of the first list again into the second but appending it, so the second list will be the sum of all the smaller list files created in my loop. The list has to be appended only if certain conditions met. It looks like something similar to this: # This is done for each log in my directory, i have a loop running for logs in mydir: for line in mylog: #...if the conditions are met list1.append(line) for item in list1: if "string" in item: #if somewhere in the list1 i have a match for a string list2.append(list1) # append every line in list1 to list2 del list1 [:] # delete the content of the list1 break else: del list1 [:] # delete the list content and start all over Does this makes sense or should I go for a different route? I need something efficient that would not take up too many cycles, since the list of logs is long and each text file is pretty big; so I thought that the lists would fit the purpose.

Output: You probably want list2.extend(list1) instead of list2.append(list1) Here's the difference: >>> a = [1, 2, 3] >>> b = [4, 5, 6] >>> c = [7, 8, 9] >>> b.append(a) >>> b [4, 5, 6, [1, 2, 3]] >>> c.extend(a) >>> c [7, 8, 9, 1, 2, 3] Since list.extend() accepts an arbitrary iterable, you can also replace for line in mylog: list1.append(line) by list1.extend(mylog)


Instruction:
Could not find a version that satisfies the requirement tensorflow

I installed the latest version of Python (3.6.4 64-bit) and the latest version of PyCharm (2017.3.3 64-bit). Then I installed some modules in PyCharm (Numpy, Pandas, etc), but when I tried installing Tensorflow it didn't install, and I got the error message: Could not find a version that satisfies the requirement TensorFlow (from versions: ) No matching distribution found for TensorFlow. Then I tried installing TensorFlow from the command prompt and I got the same error message. I did however successfully install tflearn. I also installed Python 2.7, but I got the same error message again. I googled the error and tried some of the things which were suggested to other people, but nothing worked (this included installing Flask). How can I install Tensorflow? Thanks.

Output: The latest requirements for running TensorFlow are documented in the installation documentation. TensorFlow only supports 64-bit Python TensorFlow only supports certain versions of Python (for example, Python 3.6 is not supported) So, if you're using an out-of-range version of Python (older or newer) or a 32-bit version, then you'll need to use a different version.


Instruction:
How to initialize a two-dimensional array (list of lists, if not using NumPy) in Python?

I'm beginning python and I'm trying to use a two-dimensional list, that I initially fill up with the same variable in every place. I came up with this: def initialize_twodlist(foo): twod_list = [] new = [] for i in range (0, 10): for j in range (0, 10): new.append(foo) twod_list.append(new) new = [] It gives the desired result, but feels like a workaround. Is there an easier/shorter/more elegant way to do this?

Output: A pattern that often came up in Python was bar = [] for item in some_iterable: bar.append(SOME EXPRESSION) which helped motivate the introduction of list comprehensions, which convert that snippet to bar = [SOME_EXPRESSION for item in some_iterable] which is shorter and sometimes clearer. Usually, you get in the habit of recognizing these and often replacing loops with comprehensions. Your code follows this pattern twice twod_list = [] \ for i in range (0, 10): \ new = [] \ can be replaced } this too for j in range (0, 10): } with a list / new.append(foo) / comprehension / twod_list.append(new) /


Instruction:
What does asterisk * mean in Python?

Does * have a special meaning in Python as it does in C? I saw a function like this in the Python Cookbook: def get(self, *a, **kw) Would you please explain it to me or point out where I can find an answer (Google interprets the * as wild card character and thus I cannot find a satisfactory answer).

Output: See Function Definitions in the Language Reference. If the form *identifier is present, it is initialized to a tuple receiving any excess positional parameters, defaulting to the empty tuple. If the form **identifier is present, it is initialized to a new dictionary receiving any excess keyword arguments, defaulting to a new empty dictionary. Also, see Function Calls. Assuming that one knows what positional and keyword arguments are, here are some examples: Example 1: # Excess keyword argument (python 3) example: def foo(a, b, c, **args): print("a = %s" % (a,)) print("b = %s" % (b,)) print("c = %s" % (c,)) print(args) foo(a="testa", d="excess", c="testc", b="testb", k="another_excess") As you can see in the above example, we only have parameters a, b, c in the signature of the foo function. Since d and k are not present, they are put into the args dictionary. The output of the program is: a = testa b = testb c = testc {'k': 'another_excess', 'd': 'excess'} Example 2: # Excess positional argument (python 3) example: def foo(a, b, c, *args): print("a = %s" % (a,)) print("b = %s" % (b,)) print("c = %s" % (c,)) print(args) foo("testa", "testb", "testc", "excess", "another_excess") Here, since we're testing positional arguments, the excess ones have to be on the end, and *args packs them into a tuple, so the output of this program is: a = testa b = testb c = testc ('excess', 'another_excess') You can also unpack a dictionary or a tuple into arguments of a function: def foo(a,b,c,**args): print("a=%s" % (a,)) print("b=%s" % (b,)) print("c=%s" % (c,)) print("args=%s" % (args,)) argdict = dict(a="testa", b="testb", c="testc", excessarg="string") foo(**argdict) Prints: a=testa b=testb c=testc args={'excessarg': 'string'} And def foo(a,b,c,*args): print("a=%s" % (a,)) print("b=%s" % (b,)) print("c=%s" % (c,)) print("args=%s" % (args,)) argtuple = ("testa","testb","testc","excess") foo(*argtuple) Prints: a=testa b=testb c=testc args=('excess',)


Instruction:
Format a datetime into a string with milliseconds

How can I format a datetime object as a string with milliseconds?

Output: To get a date string with milliseconds, use [:-3] to trim the last three digits of %f (microseconds): >>> from datetime import datetime >>> datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S.%f')[:-3] '2022-09-24 10:18:32.926' Or shorter: >>> from datetime import datetime >>> datetime.utcnow().strftime('%F %T.%f')[:-3] '2022-09-24 10:18:32.926' See the Python docs for more "%" format codes and the strftime(3) man page for the full list.


Instruction:
How to execute a file within the Python interpreter?

I'm trying to execute a file with Python commands from within the interpreter. I'm trying to use variables and settings from that file, not to invoke a separate process.

Output: Several ways. From the shell python someFile.py From inside IDLE, hit F5. If you're typing interactively, try this (Python3): >>> exec(open("filename.py").read()) For Python 2: >>> variables= {} >>> execfile( "someFile.py", variables ) >>> print variables # globals from the someFile module