Instruction:
Binary search (bisection) in Python

Is there a library function that performs binary search on a list/tuple and return the position of the item if found and 'False' (-1, None, etc.) if not? I found the functions bisect_left/right in the bisect module, but they still return a position even if the item is not in the list. That's perfectly fine for their intended usage, but I just want to know if an item is in the list or not (don't want to insert anything). I thought of using bisect_left and then checking if the item at that position is equal to what I'm searching, but that seems cumbersome (and I also need to do bounds checking if the number can be larger than the largest number in my list). If there is a nicer method I'd like to know about it. Edit To clarify what I need this for: I'm aware that a dictionary would be very well suited for this, but I'm trying to keep the memory consumption as low as possible. My intended usage would be a sort of double-way look-up table. I have in the table a list of values and I need to be able to access the values based on their index. And also I want to be able to find the index of a particular value or None if the value is not in the list. Using a dictionary for this would be the fastest way, but would (approximately) double the memory requirements. I was asking this question thinking that I may have overlooked something in the Python libraries. It seems I'll have to write my own code, as Moe suggested.

Output: bisect_left finds the first position p at which an element could be inserted in a given sorted range while maintaining the sorted order. That will be the position of x if x exists in the range. If p is the past-the-end position, x wasn't found. Otherwise, we can test to see if x is there to see if x was found. from bisect import bisect_left def binary_search(a, x, lo=0, hi=None): if hi is None: hi = len(a) pos = bisect_left(a, x, lo, hi) # find insertion position return pos if pos != hi and a[pos] == x else -1 # don't walk off the end


Instruction:
How to unnest (explode) a column in a pandas DataFrame, into multiple rows

I have the following DataFrame where one of the columns is an object (list type cell): df = pd.DataFrame({'A': [1, 2], 'B': [[1, 2], [1, 2]]}) Output: A B 0 1 [1, 2] 1 2 [1, 2] My expected output is: A B 0 1 1 1 1 2 3 2 1 4 2 2 What should I do to achieve this? Related question: Pandas column of lists, create a row for each list element - Good question and answer but only handle one column with list. (In my answer the self-def function will work for multiple columns, also the accepted answer is use the most time consuming apply, which is not recommended, check more info When should I (not) want to use pandas apply() in my code?)

Output: Use DataFrame.explode(): df.explode('B') A B 0 1 1 1 1 2 0 2 1 1 2 2 This was added in pandas 0.25, and support for multiple columns was added in pandas 1.3.0. [Original answer follows, including legacy solutions] I know object dtype makes the data hard to convert with pandas functions. When I received the data like this, the first thing that came to mind was to "flatten" or unnest the columns. I am using pandas and Python functions for this type of question. If you are worried about the speed of the below solutions, check out user3483203's answer, since it's using numpy and most of the time numpy is faster. I recommend Cython or numba if speed matters. Method 0: .explode() [pandas >= 0.25] [Solution moved above. The below note no longer applies since NaNs are now handled as expected, but it is kept for reference.] Given a dataframe with an empty list or a NaN in the column. An empty list will not cause an issue, but a NaN will need to be filled with a list df = pd.DataFrame({ 'A': [1, 2, 3, 4], 'B': [[1, 2], [1, 2], [], np.nan]}) df.B = df.B.fillna({i: [] for i in df.index}) # replace NaN with [] df.explode('B') A B 0 1 1 0 1 2 1 2 1 1 2 2 2 3 NaN 3 4 NaN Method 1: apply + pd.Series (easy to understand but in terms of performance not recommended.) df.set_index('A').B.apply(pd.Series).stack().reset_index(level=0).rename(columns={0:'B'}) Out[463]: A B 0 1 1 1 1 2 0 2 1 1 2 2 Method 2 Using repeat with DataFrame constructor, re-create your dataframe (good at performance, not good at multiple columns) df = pd.DataFrame({ 'A': df.A.repeat(df.B.str.len()), 'B': np.concatenate(df.B.values)}) df Out[465]: A B 0 1 1 0 1 2 1 2 1 1 2 2 Method 2.1 for example besides A we have A.1 .....A.n. If we still use the method (Method 2) above it is hard for us to re-create the columns one by one. Solution: join or merge with the index after 'unnest' the single columns s = pd.DataFrame({ 'B': np.concatenate(df.B.values)}, index=df.index.repeat(df.B.str.len())) s.join(df.drop('B', 1), how='left') Out[477]: B A 0 1 1 0 2 1 1 1 2 1 2 2 If you need the column order exactly the same as before, add reindex at the end. s.join(df.drop('B', 1), how='left').reindex(columns=df.columns) Method 3 recreate the list pd.DataFrame([[x] + [z] for x, y in df.values for z in y], columns=df.columns) Out[488]: A B 0 1 1 1 1 2 2 2 1 3 2 2 If more than two columns, use s = pd.DataFrame([[x] + [z] for x, y in zip(df.index, df.B) for z in y]) s.merge(df, left_on=0, right_index=True) Out[491]: 0 1 A B 0 0 1 1 [1, 2] 1 0 2 1 [1, 2] 2 1 1 2 [1, 2] 3 1 2 2 [1, 2] Method 4: using reindex or loc df.reindex(df.index.repeat(df.B.str.len())).assign(B=np.concatenate(df.B.values)) Out[554]: A B 0 1 1 0 1 2 1 2 1 1 2 2 #df.loc[df.index.repeat(df.B.str.len())].assign(B=np.concatenate(df.B.values)) Method 5 when the list only contains unique values: df = pd.DataFrame({'A': [1, 2], 'B': [[1, 2], [3, 4]]}) from collections import ChainMap d = dict(ChainMap(*map(dict.fromkeys, df['B'], df['A']))) pd.DataFrame(list(d.items()), columns=df.columns[::-1]) Out[574]: B A 0 1 1 1 2 1 2 3 2 3 4 2 Method 6: using numpy for high performance newvalues = np.dstack(( np.repeat(df.A.values, list(map(len, df.B.values))), np.concatenate(df.B.values))) pd.DataFrame(data=newvalues[0], columns=df.columns) A B 0 1 1 1 1 2 2 2 1 3 2 2 Method 7: Pure Python solution just for fun using base function itertools cycle and chain from itertools import cycle, chain l = df.values.tolist() l1 = [ list(zip([x[0]], cycle(x[1])) if len([x[0]]) > len(x[1]) else list(zip(cycle([x[0]]), x[1]))) for x in l] pd.DataFrame(list(chain.from_iterable(l1)), columns=df.columns) A B 0 1 1 1 1 2 2 2 1 3 2 2 Generalizing to multiple columns df = pd.DataFrame({ 'A': [1, 2], 'B': [[1, 2], [3, 4]], 'C': [[1, 2], [3, 4]]}) df Out[592]: A B C 0 1 [1, 2] [1, 2] 1 2 [3, 4] [3, 4] Self-def function: def unnesting(df, explode): idx = df.index.repeat(df[explode[0]].str.len()) df1 = pd.concat( [pd.DataFrame({x: np.concatenate(df[x].values)}) for x in explode], axis=1) df1.index = idx return df1.join(df.drop(explode, 1), how='left') unnesting(df, ['B', 'C']) Out[609]: B C A 0 1 1 1 0 2 2 1 1 3 3 2 1 4 4 2 Column-wise Unnesting All above methods are talking about vertical unnesting and explode. If you need to expand the list horizontally, try pd.DataFrame constructor df.join(pd.DataFrame(df.B.tolist(), index=df.index).add_prefix('B_')) Out[33]: A B C B_0 B_1 0 1 [1, 2] [1, 2] 1 2 1 2 [3, 4] [3, 4] 3 4 Updated function def unnesting(df, explode, axis): if axis == 1: idx = df.index.repeat(df[explode[0]].str.len()) df1 = pd.concat( [pd.DataFrame({x: np.concatenate(df[x].values)}) for x in explode], axis=1) df1.index = idx return df1.join(df.drop(explode, 1), how='left') else: df1 = pd.concat( [pd.DataFrame(df[x].tolist(), index=df.index).add_prefix(x) for x in explode], axis=1) return df1.join(df.drop(explode, 1), how='left') Test output unnesting(df, ['B', 'C'], axis=0) Out[36]: B0 B1 C0 C1 A 0 1 2 1 2 1 1 3 4 3 4 2 Update 2021-02-17 with original explode function def unnesting(df, explode, axis): if axis == 1: df1 = pd.concat( [df[x].explode() for x in explode], axis=1) return df1.join(df.drop(explode, 1), how='left') else: df1 = pd.concat( [pd.DataFrame(df[x].tolist(), index=df.index).add_prefix(x) for x in explode], axis=1) return df1.join(df.drop(explode, 1), how='left')


Instruction:
pandas unique values multiple columns

df = pd.DataFrame({'Col1': ['Bob', 'Joe', 'Bill', 'Mary', 'Joe'], 'Col2': ['Joe', 'Steve', 'Bob', 'Bob', 'Steve'], 'Col3': np.random.random(5)}) What is the best way to return the unique values of 'Col1' and 'Col2'? The desired output is 'Bob', 'Joe', 'Bill', 'Mary', 'Steve'

Output: pd.unique returns the unique values from an input array, or DataFrame column or index. The input to this function needs to be one-dimensional, so multiple columns will need to be combined. The simplest way is to select the columns you want and then view the values in a flattened NumPy array. The whole operation looks like this: >>> pd.unique(df[['Col1', 'Col2']].values.ravel('K')) array(['Bob', 'Joe', 'Bill', 'Mary', 'Steve'], dtype=object) Note that ravel() is an array method that returns a view (if possible) of a multidimensional array. The argument 'K' tells the method to flatten the array in the order the elements are stored in the memory (pandas typically stores underlying arrays in Fortran-contiguous order; columns before rows). This can be significantly faster than using the method's default 'C' order. An alternative way is to select the columns and pass them to np.unique: >>> np.unique(df[['Col1', 'Col2']].values) array(['Bill', 'Bob', 'Joe', 'Mary', 'Steve'], dtype=object) There is no need to use ravel() here as the method handles multidimensional arrays. Even so, this is likely to be slower than pd.unique as it uses a sort-based algorithm rather than a hashtable to identify unique values. The difference in speed is significant for larger DataFrames (especially if there are only a handful of unique values): >>> df1 = pd.concat([df]*100000, ignore_index=True) # DataFrame with 500000 rows >>> %timeit np.unique(df1[['Col1', 'Col2']].values) 1 loop, best of 3: 1.12 s per loop >>> %timeit pd.unique(df1[['Col1', 'Col2']].values.ravel('K')) 10 loops, best of 3: 38.9 ms per loop >>> %timeit pd.unique(df1[['Col1', 'Col2']].values.ravel()) # ravel using C order 10 loops, best of 3: 49.9 ms per loop


Instruction:
Parse a .py file, read the AST, modify it, then write back the modified source code

I want to programmatically edit python source code. Basically I want to read a .py file, generate the AST, and then write back the modified python source code (i.e. another .py file). There are ways to parse/compile python source code using standard python modules, such as ast or compiler. However, I don't think any of them support ways to modify the source code (e.g. delete this function declaration) and then write back the modifying python source code. UPDATE: The reason I want to do this is I'd like to write a Mutation testing library for python, mostly by deleting statements / expressions, rerunning tests and seeing what breaks.

Output: Pythoscope does this to the test cases it automatically generates as does the 2to3 tool for python 2.6 (it converts python 2.x source into python 3.x source). Both these tools uses the lib2to3 library which is an implementation of the python parser/compiler machinery that can preserve comments in source when it's round tripped from source -> AST -> source. The rope project may meet your needs if you want to do more refactoring like transforms. The ast module is your other option, and there's an older example of how to "unparse" syntax trees back into code (using the parser module). But the ast module is more useful when doing an AST transform on code that is then transformed into a code object. The redbaron project also may be a good fit (ht Xavier Combelle)


Instruction:
Encrypt and decrypt using PyCrypto AES-256

I'm trying to build two functions using PyCrypto that accept two parameters: the message and the key, and then encrypt/decrypt the message. I found several links on the web to help me out, but each one of them has flaws: This one at codekoala uses os.urandom, which is discouraged by PyCrypto. Moreover, the key I give to the function is not guaranteed to have the exact length expected. What can I do to make that happen? Also, there are several modes, which one is recommended? I don't know what to use :/ Finally, what exactly is the IV? Can I provide a different IV for encrypting and decrypting, or will this return in a different result?

Output: Here is my implementation, and it works for me with some fixes. It enhances the alignment of the key and secret phrase with 32 bytes and IV to 16 bytes: import base64 import hashlib from Crypto import Random from Crypto.Cipher import AES class AESCipher(object): def __init__(self, key): self.bs = AES.block_size self.key = hashlib.sha256(key.encode()).digest() def encrypt(self, raw): raw = self._pad(raw) iv = Random.new().read(AES.block_size) cipher = AES.new(self.key, AES.MODE_CBC, iv) return base64.b64encode(iv + cipher.encrypt(raw.encode())) def decrypt(self, enc): enc = base64.b64decode(enc) iv = enc[:AES.block_size] cipher = AES.new(self.key, AES.MODE_CBC, iv) return AESCipher._unpad(cipher.decrypt(enc[AES.block_size:])).decode('utf-8') def _pad(self, s): return s + (self.bs - len(s) % self.bs) * chr(self.bs - len(s) % self.bs) @staticmethod def _unpad(s): return s[:-ord(s[len(s)-1:])]


Instruction:
What is "pkg-resources==0.0.0" in output of pip freeze command

When I run pip freeze I see (among other expected packages) pkg-resources==0.0.0. I have seen a few posts mentioning this package (including this one), but none explaining what it is, or why it is included in the output of pip freeze. The main reason I am wondering is out of curiosity, but also, it seems to break things in some cases when trying to install packages with a requirements.txt file generated with pip freeze that includes the pkg-resources==0.0.0 line (for example when Travis CI tries to install dependencies through pip and finds this line). What is pkg-resources, and is it OK to remove this line from requirements.txt? Update: I have found that this line only seems to exist in the output of pip freeze when I am in a virtualenv. I am still not sure what it is or what it does, but I will investigate further knowing that it is likely related to virtualenv.

Output: According to https://github.com/pypa/pip/issues/4022, this is a bug resulting from Ubuntu providing incorrect metadata to pip. So, no there does not seem to be a good reason for this behaviour. I filed a follow-up bug with Ubuntu. https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1635463 To backup the previous answer, it should be safe to remove that line from your requirements.txt. Here is an example Make file stanza that safely freezes your package list (drop in your Makefile and run with make freeze): freeze: pip freeze | grep -v "pkg-resources" > requirements.txt edit 2022 July 06: I have been informed that the package name differs depending on the system in use (pkg-resources vs pkg_resources). Please see the comments attached to this answer for differences in usage between different versions of Debian/Ubuntu. As pkg-resources is the historically correct package name at the time this was posted (almost 6 years ago) for the system in question, it will remain unchanged in this answer.


Instruction:
Save Dataframe to csv directly to s3 Python

I have a pandas DataFrame that I want to upload to a new CSV file. The problem is that I don't want to save the file locally before transferring it to s3. Is there any method like to_csv for writing the dataframe to s3 directly? I am using boto3. Here is what I have so far: import boto3 s3 = boto3.client('s3', aws_access_key_id='key', aws_secret_access_key='secret_key') read_file = s3.get_object(Bucket, Key) df = pd.read_csv(read_file['Body']) # Make alterations to DataFrame # Then export DataFrame to CSV through direct transfer to s3

Output: You can use: from io import StringIO # python3; python2: BytesIO import boto3 bucket = 'my_bucket_name' # already created on S3 csv_buffer = StringIO() df.to_csv(csv_buffer) s3_resource = boto3.resource('s3') s3_resource.Object(bucket, 'df.csv').put(Body=csv_buffer.getvalue()) In addition, pandas now also handles reading and writing remote files via fspec, if installed. In other words, in this case df.to_csv('s3://bucket/folder/path/file.csv) also works.


Instruction:
Lambda function in list comprehensions

Why is the output of the following two list comprehensions different, even though f and the lambda function are the same? f = lambda x: x*x [f(x) for x in range(10)] and [lambda x: x*x for x in range(10)] Mind you, both type(f) and type(lambda x: x*x) return the same type.

Output: The first one creates a single lambda function and calls it ten times. The second one doesn't call the function. It creates 10 different lambda functions. It puts all of those in a list. To make it equivalent to the first you need: [(lambda x: x*x)(x) for x in range(10)] Or better yet: [x*x for x in range(10)]


Instruction:
How do I get the opposite (negation) of a Boolean in Python?

For the following sample: def fuctionName(int, bool): if int in range(...): if bool == True: return False else: return True Is there any way to skip the second if-statement? Just to tell the computer to return the opposite of the boolean bool?

Output: To negate a boolean, you can use the not operator: not bool Or in your case, the if/return blocks can be replaced by: return not bool Be sure to note the operator precedence rules, and the negated is and in operators: a is not b and a not in b.


Instruction:
How to add conda environment to jupyter lab

I'm using Jupyter Lab and I'm having trouble to add conda environment. The idea is to launch Jupyter Lab from my base environment, and then to be able to choose my other conda envs as kernels. I installed the package nb_conda_kernels which is supposed to do just that, but it's not working as I want. Indeed, let's assume I create a new Conda Environment, then I launch jupyter lab from base, I can't see the new environment as an available kernel. I have found a "fix", which works everytime but is not convenient at all. If I install Jupyter Notebook in my new environment, then launch a jupyter notebook from this new environment, close it, go back to base environment, and then launch Jupyter Lab from base environment, my new environment is available as a kernel in Jupyter Lab. If you know how to make it work without this "fix", I would be very grateful.

Output: A solution using nb_conda_kernels. First, install it in your base environment : (base)$ conda install -c conda-forge nb_conda_kernels Then in order to get a kernel for the conda_env cenv : $ conda activate cenv (cenv)$ conda install ipykernel (cenv)$ conda deactivate You will get a new kernel named Python [conda env:cenv] in your next run of jupyter lab / jupyter notebook Note : If you have installed nb_conda_kernels, and want to create a new conda environment and have it accessible right away then conda create -n new_env_name ipykernel will do the job.


Instruction:
Access nested dictionary items via a list of keys?

I have a complex dictionary structure which I would like to access via a list of keys to address the correct item. dataDict = { "a":{ "r": 1, "s": 2, "t": 3 }, "b":{ "u": 1, "v": { "x": 1, "y": 2, "z": 3 }, "w": 3 } } maplist = ["a", "r"] or maplist = ["b", "v", "y"] I have made the following code which works but I'm sure there is a better and more efficient way to do this if anyone has an idea. # Get a given data from a dictionary with position provided as a list def getFromDict(dataDict, mapList): for k in mapList: dataDict = dataDict[k] return dataDict # Set a given data in a dictionary with position provided as a list def setInDict(dataDict, mapList, value): for k in mapList[:-1]: dataDict = dataDict[k] dataDict[mapList[-1]] = value

Output: Use reduce() to traverse the dictionary: from functools import reduce # forward compatibility for Python 3 import operator def getFromDict(dataDict, mapList): return reduce(operator.getitem, mapList, dataDict) and reuse getFromDict to find the location to store the value for setInDict(): def setInDict(dataDict, mapList, value): getFromDict(dataDict, mapList[:-1])[mapList[-1]] = value All but the last element in mapList is needed to find the 'parent' dictionary to add the value to, then use the last element to set the value to the right key. Demo: >>> getFromDict(dataDict, ["a", "r"]) 1 >>> getFromDict(dataDict, ["b", "v", "y"]) 2 >>> setInDict(dataDict, ["b", "v", "w"], 4) >>> import pprint >>> pprint.pprint(dataDict) {'a': {'r': 1, 's': 2, 't': 3}, 'b': {'u': 1, 'v': {'w': 4, 'x': 1, 'y': 2, 'z': 3}, 'w': 3}} Note that the Python PEP8 style guide prescribes snake_case names for functions. The above works equally well for lists or a mix of dictionaries and lists, so the names should really be get_by_path() and set_by_path(): from functools import reduce # forward compatibility for Python 3 import operator def get_by_path(root, items): """Access a nested object in root by item sequence.""" return reduce(operator.getitem, items, root) def set_by_path(root, items, value): """Set a value in a nested object in root by item sequence.""" get_by_path(root, items[:-1])[items[-1]] = value And for completion's sake, a function to delete a key: def del_by_path(root, items): """Delete a key-value in a nested object in root by item sequence.""" del get_by_path(root, items[:-1])[items[-1]]


Instruction:
From ND to 1D arrays

Say I have an array a: a = np.array([[1,2,3], [4,5,6]]) array([[1, 2, 3], [4, 5, 6]]) I would like to convert it to a 1D array (i.e. a column vector): b = np.reshape(a, (1,np.product(a.shape))) but this returns array([[1, 2, 3, 4, 5, 6]]) which is not the same as: array([1, 2, 3, 4, 5, 6]) I can take the first element of this array to manually convert it to a 1D array: b = np.reshape(a, (1,np.product(a.shape)))[0] but this requires me to know how many dimensions the original array has (and concatenate [0]'s when working with higher dimensions) Is there a dimensions-independent way of getting a column/row vector from an arbitrary ndarray?

Output: Use np.ravel (for a 1D view) or np.ndarray.flatten (for a 1D copy) or np.ndarray.flat (for an 1D iterator): In [12]: a = np.array([[1,2,3], [4,5,6]]) In [13]: b = a.ravel() In [14]: b Out[14]: array([1, 2, 3, 4, 5, 6]) Note that ravel() returns a view of a when possible. So modifying b also modifies a. ravel() returns a view when the 1D elements are contiguous in memory, but would return a copy if, for example, a were made from slicing another array using a non-unit step size (e.g. a = x[::2]). If you want a copy rather than a view, use In [15]: c = a.flatten() If you just want an iterator, use np.ndarray.flat: In [20]: d = a.flat In [21]: d Out[21]: <numpy.flatiter object at 0x8ec2068> In [22]: list(d) Out[22]: [1, 2, 3, 4, 5, 6]


Instruction:
How to read first N lines of a file?

We have a large raw data file that we would like to trim to a specified size. How would I go about getting the first N lines of a text file in python? Will the OS being used have any effect on the implementation?

Output: Python 3: with open(path_to_file) as input_file: head = [next(input_file) for _ in range(lines_number)] print(head) Python 2: with open(path_to_file) as input_file: head = [next(input_file) for _ in xrange(lines_number)] print head Here's another way (both Python 2 & 3): from itertools import islice with open(path_to_file) as input_file: head = list(islice(input_file, lines_number)) print(head)


Instruction:
How can I get Python Requests to trust a self-signed SSL certificate?

Consider: import requests data = {'foo':'bar'} url = 'https://example.com/bar' r = requests.post(url, data=data) If the URL uses a self-signed certificate, this fails with: requests.exceptions.SSLError: [Errno 1] _ssl.c:507: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed I know that I can pass False to the verify parameter, like this: r = requests.post(url, data=data, verify=False) However, I would like to point Requests to a copy of the public key on disk and tell it to trust that certificate.

Output: Try: r = requests.post(url, data=data, verify='/path/to/public_key.pem')


Instruction:
Does SQLAlchemy have an equivalent of Django's get_or_create?

I want to get an object from the database if it already exists (based on provided parameters) or create it if it does not. Django's get_or_create (or source) does this. Is there an equivalent shortcut in SQLAlchemy? I'm currently writing it out explicitly like this: def get_or_create_instrument(session, serial_number): instrument = session.query(Instrument).filter_by(serial_number=serial_number).first() if instrument: return instrument else: instrument = Instrument(serial_number) session.add(instrument) return instrument

Output: That's basically the way to do it, there is no shortcut readily available AFAIK. You could generalize it ofcourse: def get_or_create(session, model, defaults=None, **kwargs): instance = session.query(model).filter_by(**kwargs).one_or_none() if instance: return instance, False else: params = {k: v for k, v in kwargs.items() if not isinstance(v, ClauseElement)} params.update(defaults or {}) instance = model(**params) try: session.add(instance) session.commit() except Exception: # The actual exception depends on the specific database so we catch all exceptions. This is similar to the official documentation: https://docs.sqlalchemy.org/en/latest/orm/session_transaction.html session.rollback() instance = session.query(model).filter_by(**kwargs).one() return instance, False else: return instance, True 2020 update (Python 3.9+ ONLY) Here is a cleaner version with Python 3.9's the new dict union operator (|=) def get_or_create(session, model, defaults=None, **kwargs): instance = session.query(model).filter_by(**kwargs).one_or_none() if instance: return instance, False else: kwargs |= defaults or {} instance = model(**kwargs) try: session.add(instance) session.commit() except Exception: # The actual exception depends on the specific database so we catch all exceptions. This is similar to the official documentation: https://docs.sqlalchemy.org/en/latest/orm/session_transaction.html session.rollback() instance = session.query(model).filter_by(**kwargs).one() return instance, False else: return instance, True Note: Similar to the Django version this will catch duplicate key constraints and similar errors. If your get or create is not guaranteed to return a single result it can still result in race conditions. To alleviate some of that issue you would need to add another one_or_none() style fetch right after the session.commit(). This still is no 100% guarantee against race conditions unless you also use a with_for_update() or serializable transaction mode.


Instruction:
What are the differences between feather and parquet?

Both are columnar (disk-)storage formats for use in data analysis systems. Both are integrated within Apache Arrow (pyarrow package for python) and are designed to correspond with Arrow as a columnar in-memory analytics layer. How do both formats differ? Should you always prefer feather when working with pandas when possible? What are the use cases where feather is more suitable than parquet and the other way round? Appendix I found some hints here https://github.com/wesm/feather/issues/188, but given the young age of this project, it's possibly a bit out of date. Not a serious speed test because I'm just dumping and loading a whole Dataframe but to give you some impression if you never heard of the formats before: # IPython import numpy as np import pandas as pd import pyarrow as pa import pyarrow.feather as feather import pyarrow.parquet as pq import fastparquet as fp df = pd.DataFrame({'one': [-1, np.nan, 2.5], 'two': ['foo', 'bar', 'baz'], 'three': [True, False, True]}) print("pandas df to disk ####################################################") print('example_feather:') %timeit feather.write_feather(df, 'example_feather') # 2.62 ms ± 35.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) print('example_parquet:') %timeit pq.write_table(pa.Table.from_pandas(df), 'example.parquet') # 3.19 ms ± 51 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) print() print("for comparison:") print('example_pickle:') %timeit df.to_pickle('example_pickle') # 2.75 ms ± 18.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) print('example_fp_parquet:') %timeit fp.write('example_fp_parquet', df) # 7.06 ms ± 205 µs per loop (mean ± std. dev. of 7 runs, 1 loop each) print('example_hdf:') %timeit df.to_hdf('example_hdf', 'key_to_store', mode='w', table=True) # 24.6 ms ± 4.45 ms per loop (mean ± std. dev. of 7 runs, 100 loops each) print() print("pandas df from disk ##################################################") print('example_feather:') %timeit feather.read_feather('example_feather') # 969 µs ± 1.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) print('example_parquet:') %timeit pq.read_table('example.parquet').to_pandas() # 1.9 ms ± 5.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) print("for comparison:") print('example_pickle:') %timeit pd.read_pickle('example_pickle') # 1.07 ms ± 6.21 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) print('example_fp_parquet:') %timeit fp.ParquetFile('example_fp_parquet').to_pandas() # 4.53 ms ± 260 µs per loop (mean ± std. dev. of 7 runs, 1 loop each) print('example_hdf:') %timeit pd.read_hdf('example_hdf') # 10 ms ± 43.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) # pandas version: 0.22.0 # fastparquet version: 0.1.3 # numpy version: 1.13.3 # pandas version: 0.22.0 # pyarrow version: 0.8.0 # sys.version: 3.6.3 # example Dataframe taken from https://arrow.apache.org/docs/python/parquet.html

Output: Parquet format is designed for long-term storage, where Arrow is more intended for short term or ephemeral storage (Arrow may be more suitable for long-term storage after the 1.0.0 release happens, since the binary format will be stable then) Parquet is more expensive to write than Feather as it features more layers of encoding and compression. Feather is unmodified raw columnar Arrow memory. We will probably add simple compression to Feather in the future. Due to dictionary encoding, RLE encoding, and data page compression, Parquet files will often be much smaller than Feather files Parquet is a standard storage format for analytics that's supported by many different systems: Spark, Hive, Impala, various AWS services, in future by BigQuery, etc. So if you are doing analytics, Parquet is a good option as a reference storage format for query by multiple systems The benchmarks you showed are going to be very noisy since the data you read and wrote is very small. You should try compressing at least 100MB or upwards 1GB of data to get some more informative benchmarks, see e.g. http://wesmckinney.com/blog/python-parquet-multithreading/


Instruction:
Currency formatting in Python

I am looking to format a number like 188518982.18 to £188,518,982.18 using Python. How can I do this?

Output: See the locale module. This does currency (and date) formatting. >>> import locale >>> locale.setlocale( locale.LC_ALL, '' ) 'English_United States.1252' >>> locale.currency( 188518982.18 ) '$188518982.18' >>> locale.currency( 188518982.18, grouping=True ) '$188,518,982.18'


Instruction:
How do we determine the number of days for a given month in python

I need to calculate the number of days for a given month in python. If a user inputs Feb 2011 the program should be able to tell me that Feb 2011 has 28 days. Could anyone tell me which library I should use to determine the length of a given month?

Output: You should use calendar.monthrange: >>> from calendar import monthrange >>> monthrange(2011, 2) (1, 28) Just to be clear, monthrange supports leap years as well: >>> from calendar import monthrange >>> monthrange(2012, 2) (2, 29) As @mikhail-pyrev mentions in a comment: First number is the weekday of the first day of the month, the second number is the number of days in said month.


Instruction:
"ImportError: No module named" when trying to run Python script

I'm trying to run a script that launches, amongst other things, a python script. I get a ImportError: No module named ..., however, if I launch ipython and import the same module in the same way through the interpreter, the module is accepted. What's going on, and how can I fix it? I've tried to understand how python uses PYTHONPATH but I'm thoroughly confused. Any help would greatly appreciated.

Output: This issue arises due to the ways in which the command line IPython interpreter uses your current path vs. the way a separate process does (be it an IPython notebook, external process, etc). IPython will look for modules to import that are not only found in your sys.path, but also on your current working directory. When starting an interpreter from the command line, the current directory you're operating in is the same one you started ipython in. If you run import os os.getcwd() you'll see this is true. However, let's say you're using an ipython notebook, run os.getcwd() and your current working directory is instead the folder in which you told the notebook to operate from in your ipython_notebook_config.py file (typically using the c.NotebookManager.notebook_dir setting). The solution is to provide the python interpreter with the path-to-your-module. The simplest solution is to append that path to your sys.path list. In your notebook, first try: import sys sys.path.append('my/path/to/module/folder') import module_of_interest If that doesn't work, you've got a different problem on your hands unrelated to path-to-import and you should provide more info about your problem. The better (and more permanent) way to solve this is to set your PYTHONPATH, which provides the interpreter with additional directories look in for python packages/modules. Editing or setting the PYTHONPATH as a global var is os dependent, and is discussed in detail here for Unix or Windows.


Instruction:
How could I use requests in asyncio?

I want to do parallel http request tasks in asyncio, but I find that python-requests would block the event loop of asyncio. I've found aiohttp but it couldn't provide the service of http request using a http proxy. So I want to know if there's a way to do asynchronous http requests with the help of asyncio.

Output: To use requests (or any other blocking libraries) with asyncio, you can use BaseEventLoop.run_in_executor to run a function in another thread and yield from it to get the result. For example: import asyncio import requests @asyncio.coroutine def main(): loop = asyncio.get_event_loop() future1 = loop.run_in_executor(None, requests.get, 'http://www.google.com') future2 = loop.run_in_executor(None, requests.get, 'http://www.google.co.uk') response1 = yield from future1 response2 = yield from future2 print(response1.text) print(response2.text) asyncio.run(main()) This will get both responses in parallel. With python 3.5 you can use the new await/async syntax: import asyncio import requests async def main(): loop = asyncio.get_event_loop() response1 = await loop.run_in_executor(None, requests.get, 'http://www.google.com') response2 = await loop.run_in_executor(None, requests.get, 'http://www.google.co.uk') print(response1.text) print(response2.text) asyncio.run(main()) See PEP0492 for more.


Instruction:
How to sort a list of strings numerically

I know that this sounds trivial, but I did not realize that the sort() function of Python was weird. I have a list of "numbers" that are actually in string form, so I first convert them to ints, then attempt a sort. list1=["1","10","3","22","23","4","2","200"] for item in list1: item=int(item) list1.sort() print list1 Gives me: ['1', '10', '2', '200', '22', '23', '3', '4'] I want ['1','2','3','4','10','22','23','200'] I've looked around for some of the algorithms associated with sorting numeric sets, but the ones I found all involved sorting alphanumeric sets. I know this is probably a no-brainer problem, but Google and my textbook don't offer anything more or less useful than the .sort() function.

Output: You haven't actually converted your strings to ints. Or rather, you did, but then you didn't do anything with the results. What you want is: list1 = ["1","10","3","22","23","4","2","200"] list1 = [int(x) for x in list1] list1.sort() If for some reason you need to keep strings instead of ints (usually a bad idea, but maybe you need to preserve leading zeros or something), you can use a key function. sort takes a named parameter, key, which is a function that is called on each element before it is compared. The key function's return values are compared instead of comparing the list elements directly: list1 = ["1","10","3","22","23","4","2","200"] # call int(x) on each element before comparing it list1.sort(key=int) # or if you want to do it all in the same line list1 = sorted([int(x) for x in list1])


Instruction:
Get last result in interactive Python shell

In many symbolic math systems, such as Matlab or Mathematica, you can use a variable like Ans or % to retrieve the last computed value. Is there a similar facility in the Python shell?

Output: Underscore. >>> 5+5 10 >>> _ 10 >>> _ + 5 15 >>> _ 15


Instruction:
Re-raise exception with a different type and message, preserving existing information

I'm writing a module and want to have a unified exception hierarchy for the exceptions that it can raise (e.g. inheriting from a FooError abstract class for all the foo module's specific exceptions). This allows users of the module to catch those particular exceptions and handle them distinctly, if needed. But many of the exceptions raised from the module are raised because of some other exception; e.g. failing at some task because of an OSError on a file. What I need is to “wrap” the exception caught such that it has a different type and message, so that information is available further up the propagation hierarchy by whatever catches the exception. But I don't want to lose the existing type, message, and stack trace; that's all useful information for someone trying to debug the problem. A top-level exception handler is no good, since I'm trying to decorate the exception before it makes its way further up the propagation stack, and the top-level handler is too late. This is partly solved by deriving my module foo's specific exception types from the existing type (e.g. class FooPermissionError(OSError, FooError)), but that doesn't make it any easier to wrap the existing exception instance in a new type, nor modify the message. Python's PEP 3134 “Exception Chaining and Embedded Tracebacks” discusses a change accepted in Python 3.0 for “chaining” exception objects, to indicate that a new exception was raised during the handling of an existing exception. What I'm trying to do is related: I need it also working in earlier Python versions, and I need it not for chaining, but only for polymorphism. What is the right way to do this?

Output: Python 3 introduced exception chaining (as described in PEP 3134). This allows, when raising an exception, to cite an existing exception as the “cause”: try: frobnicate() except KeyError as exc: raise ValueError("Bad grape") from exc The caught exception (exc, a KeyError) thereby becomes part of (is the “cause of”) the new exception, a ValueError. The “cause” is available to whatever code catches the new exception. By using this feature, the __cause__ attribute is set. The built-in exception handler also knows how to report the exception's “cause” and “context” along with the traceback. In Python 2, it appears this use case has no good answer (as described by Ian Bicking and Ned Batchelder). Bummer.


Instruction:
How to get all of the immediate subdirectories in Python

I'm trying to write a simple Python script that will copy a index.tpl to index.html in all of the subdirectories (with a few exceptions). I'm getting bogged down by trying to get the list of subdirectories.

Output: I did some speed testing on various functions to return the full path to all current subdirectories. tl;dr: Always use scandir: list_subfolders_with_paths = [f.path for f in os.scandir(path) if f.is_dir()] Bonus: With scandir you can also simply only get folder names by using f.name instead of f.path. This (as well as all other functions below) will not use natural sorting. This means results will be sorted like this: 1, 10, 2. To get natural sorting (1, 2, 10), please have a look at https://stackoverflow.com/a/48030307/2441026 Results: scandir is: 3x faster than walk, 32x faster than listdir (with filter), 35x faster than Pathlib and 36x faster than listdir and 37x (!) faster than glob. Scandir: 0.977 Walk: 3.011 Listdir (filter): 31.288 Pathlib: 34.075 Listdir: 35.501 Glob: 36.277 Tested with W7x64, Python 3.8.1. Folder with 440 subfolders. In case you wonder if listdir could be speed up by not doing os.path.join() twice, yes, but the difference is basically nonexistent. Code: import os import pathlib import timeit import glob path = r"<example_path>" def a(): list_subfolders_with_paths = [f.path for f in os.scandir(path) if f.is_dir()] # print(len(list_subfolders_with_paths)) def b(): list_subfolders_with_paths = [os.path.join(path, f) for f in os.listdir(path) if os.path.isdir(os.path.join(path, f))] # print(len(list_subfolders_with_paths)) def c(): list_subfolders_with_paths = [] for root, dirs, files in os.walk(path): for dir in dirs: list_subfolders_with_paths.append( os.path.join(root, dir) ) break # print(len(list_subfolders_with_paths)) def d(): list_subfolders_with_paths = glob.glob(path + '/*/') # print(len(list_subfolders_with_paths)) def e(): list_subfolders_with_paths = list(filter(os.path.isdir, [os.path.join(path, f) for f in os.listdir(path)])) # print(len(list(list_subfolders_with_paths))) def f(): p = pathlib.Path(path) list_subfolders_with_paths = [x for x in p.iterdir() if x.is_dir()] # print(len(list_subfolders_with_paths)) print(f"Scandir: {timeit.timeit(a, number=1000):.3f}") print(f"Listdir: {timeit.timeit(b, number=1000):.3f}") print(f"Walk: {timeit.timeit(c, number=1000):.3f}") print(f"Glob: {timeit.timeit(d, number=1000):.3f}") print(f"Listdir (filter): {timeit.timeit(e, number=1000):.3f}") print(f"Pathlib: {timeit.timeit(f, number=1000):.3f}")


Instruction:
Python Pandas: How to read only first n rows of CSV files in?

I have a very large data set and I can't afford to read the entire data set in. So, I'm thinking of reading only one chunk of it to train but I have no idea how to do it.

Output: If you only want to read the first 999,999 (non-header) rows: read_csv(..., nrows=999999) If you only want to read rows 1,000,000 ... 1,999,999 read_csv(..., skiprows=1000000, nrows=999999) nrows : int, default None Number of rows of file to read. Useful for reading pieces of large files* skiprows : list-like or integer Row numbers to skip (0-indexed) or number of rows to skip (int) at the start of the file and for large files, you'll probably also want to use chunksize: chunksize : int, default None Return TextFileReader object for iteration pandas.io.parsers.read_csv documentation


Instruction:
What's the difference between lists enclosed by square brackets and parentheses in Python?

>>> x=[1,2] >>> x[1] 2 >>> x=(1,2) >>> x[1] 2 Are they both valid? Is one preferred for some reason?

Output: Square brackets are lists while parentheses are tuples. A list is mutable, meaning you can change its contents: >>> x = [1,2] >>> x.append(3) >>> x [1, 2, 3] while tuples are not: >>> x = (1,2) >>> x (1, 2) >>> x.append(3) Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'tuple' object has no attribute 'append' The other main difference is that a tuple is hashable, meaning that you can use it as a key to a dictionary, among other things. For example: >>> x = (1,2) >>> y = [1,2] >>> z = {} >>> z[x] = 3 >>> z {(1, 2): 3} >>> z[y] = 4 Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: unhashable type: 'list' Note that, as many people have pointed out, you can add tuples together. For example: >>> x = (1,2) >>> x += (3,) >>> x (1, 2, 3) However, this does not mean tuples are mutable. In the example above, a new tuple is constructed by adding together the two tuples as arguments. The original tuple is not modified. To demonstrate this, consider the following: >>> x = (1,2) >>> y = x >>> x += (3,) >>> x (1, 2, 3) >>> y (1, 2) Whereas, if you were to construct this same example with a list, y would also be updated: >>> x = [1, 2] >>> y = x >>> x += [3] >>> x [1, 2, 3] >>> y [1, 2, 3]


Instruction:
Pretty-Print JSON Data to a File using Python

A project for class involves parsing Twitter JSON data. I'm getting the data and setting it to the file without much trouble, but it's all in one line. This is fine for the data manipulation I'm trying to do, but the file is ridiculously hard to read and I can't examine it very well, making the code writing for the data manipulation part very difficult. Does anyone know how to do that from within Python (i.e. not using the command line tool, which I can't get to work)? Here's my code so far: header, output = client.request(twitterRequest, method="GET", body=None, headers=None, force_auth_header=True) # now write output to a file twitterDataFile = open("twitterData.json", "wb") # magic happens here to make it pretty-printed twitterDataFile.write(output) twitterDataFile.close() Note I appreciate people pointing me to simplejson documentation and such, but as I have stated, I have already looked at that and continue to need assistance. A truly helpful reply will be more detailed and explanatory than the examples found there. Thanks Also: Trying this in the windows command line: more twitterData.json | python -mjson.tool > twitterData-pretty.json results in this: Invalid control character at: line 1 column 65535 (char 65535) I'd give you the data I'm using, but it's very large and you've already seen the code I used to make the file.

Output: You should use the optional argument indent. header, output = client.request(twitterRequest, method="GET", body=None, headers=None, force_auth_header=True) # now write output to a file with open("twitterData.json", "w") as twitterDataFile: # magic happens here to make it pretty-printed twitterDataFile.write( simplejson.dumps(simplejson.loads(output), indent=4, sort_keys=True) )


Instruction:
"asyncio.run() cannot be called from a running event loop" when using Jupyter Notebook

I would like to use asyncio to get webpage html. I run the following code in jupyter notebook: import aiofiles import aiohttp from aiohttp import ClientSession async def get_info(url, session): resp = await session.request(method="GET", url=url) resp.raise_for_status() html = await resp.text(encoding='GB18030') with open('test_asyncio.html', 'w', encoding='utf-8-sig') as f: f.write(html) return html async def main(urls): async with ClientSession() as session: tasks = [get_info(url, session) for url in urls] return await asyncio.gather(*tasks) if __name__ == "__main__": url = ['http://huanyuntianxiazh.fang.com/house/1010123799/housedetail.htm', 'http://zhaoshangyonghefu010.fang.com/house/1010126863/housedetail.htm'] result = asyncio.run(main(url)) However, it returns RuntimeError: asyncio.run() cannot be called from a running event loop What is the problem? How to solve it?

Output: The asyncio.run() documentation says: This function cannot be called when another asyncio event loop is running in the same thread. In your case, jupyter (IPython ≥ 7.0) is already running an event loop: You can now use async/await at the top level in the IPython terminal and in the notebook, it should — in most of the cases — “just work”. Update IPython to version 7+, IPykernel to version 5+, and you’re off to the races. Therefore you don't need to start the event loop yourself and can instead call await main(url) directly, even if your code lies outside any asynchronous function. Modern Jupyter lab/notebook Use the following for newer versions of Jupyter (IPython ≥ 7.0): async def main(): print(1) await main() Python or older IPython If you are using Python ≥ 3.7 or IPython < 7.0, use the following: import asyncio async def main(): print(1) asyncio.run(main()) That's also that form you should use if you are running this in a python REPL or in an independent script (a bot, a web scrapper, etc.). If you are using an older version of python (< 3.7), the API to run asynchronous code was a bit less elegant: import asyncio async def main(): print(1) loop = asyncio.get_event_loop() loop.run_until_complete(hello_world()) Using await in your code In your case, you can call await main(url) as follows: url = ['url1', 'url2'] result = await main(url) for text in result: pass # text contains your html (text) response This change to recent versions of IPython makes notebook code simpler and more intuitive for beginers. Further notices Few remarks that might help you in different use cases. Jupyter vs. IPython caution There is a slight difference on how Jupyter uses the loop compared to IPython. [...] IPykernel having a persistent asyncio loop running, while Terminal IPython starts and stops a loop for each code block. This can lead to unexpected issues. Google Colab In the past, Google colab required you to do more complex loop manipulations like presented in some other answers here. Now plain await main() should just work like in IPython ≥ 7.0 (tested on Colab version 2023/08/18). Python REPL You can also run the python REPL using the asyncio concurrent context. As explained in asyncio's documentation: $ python -m asyncio asyncio REPL ... Use "await" directly instead of "asyncio.run()". >>> import asyncio >>> await asyncio.sleep(10, result='hello') 'hello' The asyncio REPL should be available for python ≥ 3.8.1. When does asyncio.run matters and why? Older versions of IPython were running in a synchronous context, which is why calling asyncio.run was mandatory. The asyncio.run function allows to run asynchronous code from a synchronous context by doing the following: starts an event loop, runs the async function passed as argument in this (new) event loop, stops the event loop once the function returned In more technical terms (notice how the argument function is called a coroutine): This function runs the passed coroutine, taking care of managing the asyncio event loop, finalizing asynchronous generators, and closing the threadpool. What happen when using await in synchronous context? If you happen to use await in a synchronous context you would get the one of the following errors: SyntaxError: 'await' outside function SyntaxError: 'await' outside async function In that case that means you need to use asyncio.run(main()) instead of await main().


Instruction:
How to solve ReadTimeoutError: HTTPSConnectionPool(host='pypi.python.org', port=443) with pip?

I recently need to install some packages pip install future pip install scikit-learn pip install numpy pip install scipy I also tried by writin sudo before them but all it came up with the following errors in red lines: Exception: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main status = self.run(options, args) File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 290, in run requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle) File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1198, in prepare_files do_download, File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1376, in unpack_url self.session, File "/usr/lib/python2.7/dist-packages/pip/download.py", line 572, in unpack_http_url download_hash = _download_url(resp, link, temp_location) File "/usr/lib/python2.7/dist-packages/pip/download.py", line 433, in _download_url for chunk in resp_read(4096): File "/usr/lib/python2.7/dist-packages/pip/download.py", line 421, in resp_read chunk_size, decode_content=False): File "/usr/lib/python2.7/dist-packages/urllib3/response.py", line 256, in stream data = self.read(amt=amt, decode_content=decode_content) File "/usr/lib/python2.7/dist-packages/urllib3/response.py", line 201, in read raise ReadTimeoutError(self._pool, None, 'Read timed out.') ReadTimeoutError: HTTPSConnectionPool(host='pypi.python.org', port=443): Read timed out. Storing debug log for failure in /root/.pip/pip.log'

Output: Use --default-timeout=100 parameter with the install: sudo pip install --default-timeout=100 future


Instruction:
Plotting time on the independent axis

I have an array of timestamps in the format (HH:MM:SS.mmmmmm) and another array of floating point numbers, each corresponding to a value in the timestamp array. Can I plot time on the x axis and the numbers on the y-axis using Matplotlib? I was trying to, but somehow it was only accepting arrays of floats. How can I get it to plot the time? Do I have to modify the format in any way?

Output: Update: This answer is outdated since matplotlib version 3.5. The plot function now handles datetime data directly. See https://matplotlib.org/3.5.1/api/_as_gen/matplotlib.pyplot.plot_date.html The use of plot_date is discouraged. This method exists for historic reasons and may be deprecated in the future. datetime-like data should directly be plotted using plot. If you need to plot plain numeric data as Matplotlib date format or need to set a timezone, call ax.xaxis.axis_date / ax.yaxis.axis_date before plot. See Axis.axis_date. Old, outdated answer: You must first convert your timestamps to Python datetime objects (use datetime.strptime). Then use date2num to convert the dates to matplotlib format. Plot the dates and values using plot_date: import matplotlib.pyplot as plt import matplotlib.dates from datetime import datetime x_values = [datetime(2021, 11, 18, 12), datetime(2021, 11, 18, 14), datetime(2021, 11, 18, 16)] y_values = [1.0, 3.0, 2.0] dates = matplotlib.dates.date2num(x_values) plt.plot_date(dates, y_values)


Instruction:
Execution of Python code with -m option or not

The python interpreter has -m module option that "Runs library module module as a script". With this python code a.py: if __name__ == "__main__": print __package__ print __name__ I tested python -m a to get "" <-- Empty String __main__ whereas python a.py returns None <-- None __main__ To me, those two invocation seems to be the same except __package__ is not None when invoked with -m option. Interestingly, with python -m runpy a, I get the same as python -m a with python module compiled to get a.pyc. What's the (practical) difference between these invocations? Any pros and cons between them? Also, David Beazley's Python Essential Reference explains it as "The -m option runs a library module as a script which executes inside the __main__ module prior to the execution of the main script". What does it mean?

Output: When you use the -m command-line flag, Python will import a module or package for you, then run it as a script. When you don't use the -m flag, the file you named is run as just a script. The distinction is important when you try to run a package. There is a big difference between: python foo/bar/baz.py and python -m foo.bar.baz as in the latter case, foo.bar is imported and relative imports will work correctly with foo.bar as the starting point. Demo: $ mkdir -p test/foo/bar $ touch test/foo/__init__.py $ touch test/foo/bar/__init__.py $ cat << EOF > test/foo/bar/baz.py > if __name__ == "__main__": > print __package__ > print __name__ > > EOF $ PYTHONPATH=test python test/foo/bar/baz.py None __main__ $ PYTHONPATH=test python -m foo.bar.baz foo.bar __main__ As a result, Python has to actually care about packages when using the -m switch. A normal script can never be a package, so __package__ is set to None. But run a package or module inside a package with -m and now there is at least the possibility of a package, so the __package__ variable is set to a string value; in the above demonstration it is set to 'foo.bar', for plain modules not inside a package it is set to an empty string. As for the __main__ module, Python imports scripts being run as it would import regular modules. A new module object is created to hold the global namespace and is stored in sys.modules['__main__']. This is what the __name__ variable refers to, it is a key in that structure. For packages, you can create a __main__.py module inside and have that run when running python -m package_name; in fact that is the only way you can run a package as a script: $ PYTHONPATH=test python -m foo.bar python: No module named foo.bar.__main__; 'foo.bar' is a package and cannot be directly executed $ cp test/foo/bar/baz.py test/foo/bar/__main__.py $ PYTHONPATH=test python -m foo.bar foo.bar __main__ So, when naming a package for running with -m, Python looks for a __main__ module contained in that package and executes that as a script. Its name is then still set to '__main__' and the module object is still stored in sys.modules['__main__'].


Instruction:
Why does Pycharm's inspector complain about "d = {}"?

When initializing a dictionary with d = {} Pycharm's code inspector generates a warning, saying This dictionary creation could be rewritten as a dictionary literal. If I rewrite it d = dict() the warning goes away. Since {} already is a dictionary literal, I'm pretty sure the message is erroneous. Furthermore, it seems like both d = {} and d = dict() are valid and Pythonic. This related question seems to conclude that the choice is just a matter of style/preference: differences between "d = dict()" and "d = {}" Why would Pycharm complain about d = {}? UPDATE: Mac nailed it. The warning actually applied to multiple lines, not just the one that was flagged. Pycharm seems to look for a sequence of consecutive statements where you initialize a dictionary and then set values in the dictionary. For example, this will trigger the warning: d = {} d['a'] = 1 But this code will not: d = {} pass d['a'] = 1

Output: What is the code following your dictionary declaration? I think PyCharm will trigger the error if you have something like: dic = {} dic['aaa'] = 5 as you could have written dic = {'aaa': 5} Note: The fact that the error goes away if you use the function dict(). This doesn't necessarily mean that pycharm believes dict() is a literal. It could just mean that it doesn't complain about it: dic = dict() dic['aaa'] = 5


Instruction:
Difference between Python datetime vs time modules

I am trying to figure out the differences between the datetime and time modules, and what each should be used for. I know that datetime provides both dates and time. What is the use of the time module? Examples would be appreciated and differences concerning timezones would especially be of interest.

Output: The time module is principally for working with Unix time stamps; expressed as a floating point number taken to be seconds since the Unix epoch. the datetime module can support many of the same operations, but provides a more object oriented set of types, and also has some limited support for time zones.


Instruction:
How to save and load cookies using Python + Selenium WebDriver

How can I save all cookies in Python's Selenium WebDriver to a .txt file, and then load them later? The documentation doesn't say much of anything about the getCookies function.

Output: You can save the current cookies as a Python object using pickle. For example: import pickle import selenium.webdriver driver = selenium.webdriver.Firefox() driver.get("http://www.google.com") pickle.dump(driver.get_cookies(), open("cookies.pkl", "wb")) And later to add them back: import pickle import selenium.webdriver driver = selenium.webdriver.Firefox() driver.get("http://www.google.com") cookies = pickle.load(open("cookies.pkl", "rb")) for cookie in cookies: driver.add_cookie(cookie)


Instruction:
How to open every file in a folder

I have a python script parse.py, which in the script open a file, say file1, and then do something maybe print out the total number of characters. filename = 'file1' f = open(filename, 'r') content = f.read() print filename, len(content) Right now, I am using stdout to direct the result to my output file - output python parse.py >> output However, I don't want to do this file by file manually, is there a way to take care of every single file automatically? Like ls | awk '{print}' | python parse.py >> output Then the problem is how could I read the file name from standardin? or there are already some built-in functions to do the ls and those kind of work easily? Thanks!

Output: Os You can list all files in the current directory using os.listdir: import os for filename in os.listdir(os.getcwd()): with open(os.path.join(os.getcwd(), filename), 'r') as f: # open in readonly mode # do your stuff Glob Or you can list only some files, depending on the file pattern using the glob module: import os, glob for filename in glob.glob('*.txt'): with open(os.path.join(os.getcwd(), filename), 'r') as f: # open in readonly mode # do your stuff It doesn't have to be the current directory you can list them in any path you want: import os, glob path = '/some/path/to/file' for filename in glob.glob(os.path.join(path, '*.txt')): with open(os.path.join(os.getcwd(), filename), 'r') as f: # open in readonly mode # do your stuff Pipe Or you can even use the pipe as you specified using fileinput import fileinput for line in fileinput.input(): # do your stuff And you can then use it with piping: ls -1 | python parse.py


Instruction:
Conda command is not recognized on Windows 10

I installed Anaconda 4.4.0 (Python 3.6 version) on Windows 10 by following the instructions here: https://www.continuum.io/downloads. However, when I open the Command prompt window and try to write conda list I get the 'conda' command is not recognized... error. I tried to run set PATH=%PATH%;C:\Users\Alex\Anaconda3 but it didn't help. I also read that I might need to edit my .bashrc file, but I don't know how to access this file, and how I should edit it.

Output: In Windows, you will have to set the path to the location where you installed Anaconda3 to. For me, I installed anaconda3 into C:\Anaconda3. Therefore you need to add C:\Anaconda3 as well as C:\Anaconda3\Scripts\ to your path variable, e.g. set PATH=%PATH%;C:\Anaconda3;C:\Anaconda3\Scripts\. You can do this via powershell (see above, https://learn.microsoft.com/en-us/windows/win32/shell/user-environment-variables ), or hit the windows key → enter environment → choose from settings → edit environment variables for your account → select Path variable → Edit → New. To test it, open a new dos shell, and you should be able to use conda commands now. E.g., try conda --version.


Instruction:
How to sort mongodb with pymongo

I'm trying to use the sort feature when querying my mongoDB, but it is failing. The same query works in the MongoDB console but not here. Code is as follows: import pymongo from pymongo import Connection connection = Connection() db = connection.myDB print db.posts.count() for post in db.posts.find({}, {'entities.user_mentions.screen_name':1}).sort({u'entities.user_mentions.screen_name':1}): print post The error I get is as follows: Traceback (most recent call last): File "find_ow.py", line 7, in <module> for post in db.posts.find({}, {'entities.user_mentions.screen_name':1}).sort({'entities.user_mentions.screen_name':1},1): File "/Library/Python/2.6/site-packages/pymongo-2.0.1-py2.6-macosx-10.6-universal.egg/pymongo/cursor.py", line 430, in sort File "/Library/Python/2.6/site-packages/pymongo-2.0.1-py2.6-macosx-10.6-universal.egg/pymongo/helpers.py", line 67, in _index_document TypeError: first item in each key pair must be a string I found a link elsewhere that says I need to place a 'u' infront of the key if using pymongo, but that didn't work either. Anyone else get this to work or is this a bug.

Output: .sort(), in pymongo, takes key and direction as parameters. So if you want to sort by, let's say, id then you should .sort("_id", 1) For multiple fields: .sort([("field1", pymongo.ASCENDING), ("field2", pymongo.DESCENDING)])


Instruction:
Excluding directories in os.walk

I'm writing a script that descends into a directory tree (using os.walk()) and then visits each file matching a certain file extension. However, since some of the directory trees that my tool will be used on also contain sub directories that in turn contain a LOT of useless (for the purpose of this script) stuff, I figured I'd add an option for the user to specify a list of directories to exclude from the traversal. This is easy enough with os.walk(). After all, it's up to me to decide whether I actually want to visit the respective files / dirs yielded by os.walk() or just skip them. The problem is that if I have, for example, a directory tree like this: root-- | --- dirA | --- dirB | --- uselessStuff -- | --- moreJunk | --- yetMoreJunk and I want to exclude uselessStuff and all its children, os.walk() will still descend into all the (potentially thousands of) sub directories of uselessStuff, which, needless to say, slows things down a lot. In an ideal world, I could tell os.walk() to not even bother yielding any more children of uselessStuff, but to my knowledge there is no way of doing that (is there?). Does anyone have an idea? Maybe there's a third-party library that provides something like that?

Output: Modifying dirs in-place will prune the (subsequent) files and directories visited by os.walk: # exclude = set(['New folder', 'Windows', 'Desktop']) for root, dirs, files in os.walk(top, topdown=True): dirs[:] = [d for d in dirs if d not in exclude] From help(os.walk): When topdown is true, the caller can modify the dirnames list in-place (e.g., via del or slice assignment), and walk will only recurse into the subdirectories whose names remain in dirnames; this can be used to prune the search...


Instruction:
No module named 'distutils.util' ...but distutils is installed?

I was wanting to upgrade my Python version (to 3.10 in this case), so after installing Python 3.10, I proceeded to try adding some modules I use, e.g., opencv, which ran into: python3.10 -m pip install opencv-python Output: Traceback (most recent call last): File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/usr/lib/python3/dist-packages/pip/__main__.py", line 16, in <module> from pip._internal.cli.main import main as _main # isort:skip # noqa File "/usr/lib/python3/dist-packages/pip/_internal/cli/main.py", line 10, in <module> from pip._internal.cli.autocompletion import autocomplete File "/usr/lib/python3/dist-packages/pip/_internal/cli/autocompletion.py", line 9, in <module> from pip._internal.cli.main_parser import create_main_parser File "/usr/lib/python3/dist-packages/pip/_internal/cli/main_parser.py", line 7, in <module> from pip._internal.cli import cmdoptions File "/usr/lib/python3/dist-packages/pip/_internal/cli/cmdoptions.py", line 19, in <module> from distutils.util import strtobool ModuleNotFoundError: No module named 'distutils.util' And sudo apt-get install python3-distutils Output: [sudo] password for jeremy: Reading package lists... Done Building dependency tree Reading state information... Done python3-distutils is already the newest version (3.8.10-0ubuntu1~20.04). ... 0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded. Since distutils already seems to be installed, I can't grok how to proceed.

Output: It looks like distutils has versioning, so after the following, it seems to be able to proceed. sudo apt-get install python3.10-distutils Output: Reading package lists... Done Building dependency tree Reading state information... Done ... Setting up python3.10-lib2to3 (3.10.0-1+focal1) ... Setting up python3.10-distutils (3.10.0-1+focal1) ... And: python3.10 -m pip install opencv-python


Instruction:
Using NumPy to build an array of all combinations of two arrays

I'm trying to run over the parameters space of a six-parameter function to study its numerical behavior before trying to do anything complex with it, so I'm searching for an efficient way to do this. My function takes float values given in a 6-dim NumPy array as input. What I tried to do initially was this: First, I created a function that takes two arrays and generate an array with all combinations of values from the two arrays: from numpy import * def comb(a, b): c = [] for i in a: for j in b: c.append(r_[i,j]) return c Then, I used reduce() to apply that to m copies of the same array: def combs(a, m): return reduce(comb, [a]*m) Finally, I evaluate my function like this: values = combs(np.arange(0, 1, 0.1), 6) for val in values: print F(val) This works, but it's way too slow. I know the space of parameters is huge, but this shouldn't be so slow. I have only sampled 106 (a million) points in this example and it took more than 15 seconds just to create the array values. Is there a more efficient way of doing this with NumPy? I can modify the way the function F takes its arguments if it's necessary.

Output: In newer versions of NumPy (>1.8.x), numpy.meshgrid() provides a much faster implementation: For pv's solution: In [113]: %timeit cartesian(([1, 2, 3], [4, 5], [6, 7])) 10000 loops, best of 3: 135 µs per loop In [114]: cartesian(([1, 2, 3], [4, 5], [6, 7])) Out[114]: array([[1, 4, 6], [1, 4, 7], [1, 5, 6], [1, 5, 7], [2, 4, 6], [2, 4, 7], [2, 5, 6], [2, 5, 7], [3, 4, 6], [3, 4, 7], [3, 5, 6], [3, 5, 7]]) numpy.meshgrid() used to be two-dimensional only, but now it is capable of being multidimensional. In this case, three-dimensional: In [115]: %timeit np.array(np.meshgrid([1, 2, 3], [4, 5], [6, 7])).T.reshape(-1,3) 10000 loops, best of 3: 74.1 µs per loop In [116]: np.array(np.meshgrid([1, 2, 3], [4, 5], [6, 7])).T.reshape(-1,3) Out[116]: array([[1, 4, 6], [1, 5, 6], [2, 4, 6], [2, 5, 6], [3, 4, 6], [3, 5, 6], [1, 4, 7], [1, 5, 7], [2, 4, 7], [2, 5, 7], [3, 4, 7], [3, 5, 7]]) Note that the order of the final result is slightly different.


Instruction:
What does %s mean in a Python format string?

What does %s mean in Python? And what does the following bit of code do? For instance... if len(sys.argv) < 2: sys.exit('Usage: %s database-name' % sys.argv[0]) if not os.path.exists(sys.argv[1]): sys.exit('ERROR: Database %s was not found!' % sys.argv[1])

Output: It is a string formatting syntax (which it borrows from C). Please see "PyFormat": Python supports formatting values into strings. Although this can include very complicated expressions, the most basic usage is to insert values into a string with the %s placeholder. Here is a really simple example: #Python 2 name = raw_input("who are you? ") print "hello %s" % (name,) #Python 3+ name = input("who are you? ") print("hello %s" % (name,)) The %s token allows me to insert (and potentially format) a string. Notice that the %s token is replaced by whatever I pass to the string after the % symbol. Notice also that I am using a tuple here as well (when you only have one string using a tuple is optional) to illustrate that multiple strings can be inserted and formatted in one statement.


Instruction:
How can we force naming of parameters when calling a function?

In Python you may have a function definition: def info(obj, spacing=10, collapse=1) which could be called in any of the following ways: info(odbchelper) info(odbchelper, 12) info(odbchelper, collapse=0) info(spacing=15, object=odbchelper) thanks to Python's allowing of any-order arguments, so long as they're named. The problem we're having is as some of our larger functions grow, people might be adding parameters between spacing and collapse, meaning that the wrong values may be going to parameters that aren't named. In addition sometimes it's not always clear as to what needs to go in. How can we force people to name certain parameters - not just a coding standard, but ideally a flag or pydev plugin? So that in the above 4 examples, only the last would pass the check as all the parameters are named.

Output: In Python 3 - Yes, you can specify * in the argument list. From docs: Parameters after “*” or “*identifier” are keyword-only parameters and may only be passed used keyword arguments. >>> def foo(pos, *, forcenamed): ... print(pos, forcenamed) ... >>> foo(pos=10, forcenamed=20) 10 20 >>> foo(10, forcenamed=20) 10 20 >>> foo(10, 20) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: foo() takes exactly 1 positional argument (2 given) This can also be combined with **kwargs: def foo(pos, *, forcenamed, **kwargs): To complete example: def foo(pos, *, forcenamed ): print(pos, forcenamed) foo(pos=10, forcenamed=20) foo(10, forcenamed=20) # basically you always have to give the value! foo(10) output: Traceback (most recent call last): File "/Users/brando/anaconda3/envs/metalearning/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3444, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "<ipython-input-12-ab74191b3e9e>", line 7, in <module> foo(10) TypeError: foo() missing 1 required keyword-only argument: 'forcenamed' So you are forced to always give the value. If you don't call it you don't have to do anything else named argument forced.


Instruction:
How do I disable "missing docstring" warnings at a file-level in Pylint?

Pylint throws errors that some of the files are missing docstrings. I try and add docstrings to each class, method and function, but it seems that Pylint also checks that files should have a docstring at the beginning of them. Can I disable this somehow? I would like to be notified of a docstring is missing inside a class, function or method, but it shouldn't be mandatory for a file to have a docstring. (Is there a term for the legal jargon often found at the beginning of a proprietary source file? Any examples? I don't know whether it is a okay to post such a trivial question separately.)

Output: It is nice for a Python module to have a docstring, explaining what the module does, what it provides, examples of how to use the classes. This is different from the comments that you often see at the beginning of a file giving the copyright and license information, which IMO should not go in the docstring (some even argue that they should disappear altogether, see e.g. Get Rid of Source Code Templates) With Pylint 2.4 and above, you can differentiate between the various missing-docstring by using the three following sub-messages: C0114 (missing-module-docstring) C0115 (missing-class-docstring) C0116 (missing-function-docstring) So the following .pylintrc file should work: [MASTER] disable= C0114, # missing-module-docstring For previous versions of Pylint, it does not have a separate code for the various place where docstrings can occur, so all you can do is disable C0111. The problem is that if you disable this at module scope, then it will be disabled everywhere in the module (i.e., you won't get any C line for missing function / class / method docstring. Which arguably is not nice. So I suggest adding that small missing docstring, saying something like: """ high level support for doing this and that. """ Soon enough, you'll be finding useful things to put in there, such as providing examples of how to use the various classes / functions of the module which do not necessarily belong to the individual docstrings of the classes / functions (such as how these interact, or something like a quick start guide).


Instruction:
How to test single file under pytest

How do you test a single file in pytest? I could only find ignore options and no "test this file only" option in the docs. Preferably this would work on the command line instead of setup.cfg, as I would like to run different file tests in the ide. The entire suite takes too long.

Output: simply run pytest with the path to the file something like pytest tests/test_file.py Use the :: syntax to run a specific test in the test file: pytest test_mod.py::test_func Here test_func can be a test method or a class (e.g.: pytest test_mod.py::TestClass). For more ways and details, see "Specifying which tests to run" in the docs.


Instruction:
How do you use the ellipsis slicing syntax in Python?

This came up in Hidden features of Python, but I can't see good documentation or examples that explain how the feature works.

Output: Ellipsis, or ... is not a hidden feature, it's just a constant. It's quite different to, say, javascript ES6 where it's a part of the language syntax. No builtin class or Python language constuct makes use of it. So the syntax for it depends entirely on you, or someone else, having written code to understand it. Numpy uses it, as stated in the documentation. Some examples here. In your own class, you'd use it like this: >>> class TestEllipsis(object): ... def __getitem__(self, item): ... if item is Ellipsis: ... return "Returning all items" ... else: ... return "return %r items" % item ... >>> x = TestEllipsis() >>> print x[2] return 2 items >>> print x[...] Returning all items Of course, there is the python documentation, and language reference. But those aren't very helpful.


Instruction:
Python xml ElementTree from a string source?

The ElementTree.parse reads from a file, how can I use this if I already have the XML data in a string? Maybe I am missing something here, but there must be a way to use the ElementTree without writing out the string to a file and reading it again. xml.etree.elementtree

Output: If you're using xml.etree.ElementTree.parse to parse from a file, then you can use xml.etree.ElementTree.fromstring to get the root Element of the document. Often you don't actually need an ElementTree. See xml.etree.ElementTree


Instruction:
Combine several images horizontally with Python

I am trying to horizontally combine some JPEG images in Python. Problem I have 3 images - each is 148 x 95 - see attached. I just made 3 copies of the same image - that is why they are the same. My attempt I am trying to horizontally join them using the following code: from PIL import Image list_im = ['Test1.jpg','Test2.jpg','Test3.jpg'] # creates a new empty image, RGB mode, and size 444 by 95 new_im = Image.new('RGB', (444,95)) for elem in list_im: for i in xrange(0,444,95): im=Image.open(elem) new_im.paste(im, (i,0)) new_im.save('test.jpg') However, this is producing the output attached as test.jpg. Question Is there a way to horizontally concatenate these images such that the sub-images in test.jpg do not have an extra partial image showing? Additional Information I am looking for a way to horizontally concatenate n images. I would like to use this code generally so I would prefer to: not to hard-code image dimensions, if possible specify dimensions in one line so that they can be easily changed

Output: You can do something like this: from PIL import Image images = [Image.open(x) for x in ['Test1.jpg', 'Test2.jpg', 'Test3.jpg']] widths, heights = zip(*(i.size for i in images)) total_width = sum(widths) max_height = max(heights) new_im = Image.new('RGB', (total_width, max_height)) x_offset = 0 for im in images: new_im.paste(im, (x_offset,0)) x_offset += im.size[0] new_im.save('test.jpg') Test1.jpg Test2.jpg Test3.jpg test.jpg The nested for for i in xrange(0,444,95): is pasting each image 5 times, staggered 95 pixels apart. Each outer loop iteration pasting over the previous. for elem in list_im: for i in xrange(0,444,95): im=Image.open(elem) new_im.paste(im, (i,0)) new_im.save('new_' + elem + '.jpg')


Instruction:
APT command line interface-like yes/no input?

Is there any short way to achieve what the APT (Advanced Package Tool) command line interface does in Python? I mean, when the package manager prompts a yes/no question followed by [Yes/no], the script accepts YES/Y/yes/y or Enter (defaults to Yes as hinted by the capital letter). The only thing I find in the official docs is input and raw_input... I know it's not that hard to emulate, but it's annoying to rewrite :|

Output: As you mentioned, the easiest way is to use raw_input() (or simply input() for Python 3). There is no built-in way to do this. From Recipe 577058: import sys def query_yes_no(question, default="yes"): """Ask a yes/no question via raw_input() and return their answer. "question" is a string that is presented to the user. "default" is the presumed answer if the user just hits <Enter>. It must be "yes" (the default), "no" or None (meaning an answer is required of the user). The "answer" return value is True for "yes" or False for "no". """ valid = {"yes": True, "y": True, "ye": True, "no": False, "n": False} if default is None: prompt = " [y/n] " elif default == "yes": prompt = " [Y/n] " elif default == "no": prompt = " [y/N] " else: raise ValueError("invalid default answer: '%s'" % default) while True: sys.stdout.write(question + prompt) choice = input().lower() if default is not None and choice == "": return valid[default] elif choice in valid: return valid[choice] else: sys.stdout.write("Please respond with 'yes' or 'no' " "(or 'y' or 'n').\n") (For Python 2, use raw_input instead of input.) Usage example: >>> query_yes_no("Is cabbage yummier than cauliflower?") Is cabbage yummier than cauliflower? [Y/n] oops Please respond with 'yes' or 'no' (or 'y' or 'n'). Is cabbage yummier than cauliflower? [Y/n] [ENTER] >>> True >>> query_yes_no("Is cabbage yummier than cauliflower?", None) Is cabbage yummier than cauliflower? [y/n] [ENTER] Please respond with 'yes' or 'no' (or 'y' or 'n'). Is cabbage yummier than cauliflower? [y/n] y >>> True


Instruction:
Check the total number of parameters in a PyTorch model

How do I count the total number of parameters in a PyTorch model? Something similar to model.count_params() in Keras.

Output: PyTorch doesn't have a function to calculate the total number of parameters as Keras does, but it's possible to sum the number of elements for every parameter group: pytorch_total_params = sum(p.numel() for p in model.parameters()) If you want to calculate only the trainable parameters: pytorch_total_params = sum(p.numel() for p in model.parameters() if p.requires_grad) Answer inspired by this answer on PyTorch Forums.


Instruction:
Find and replace string values in list

I got this list: words = ['how', 'much', 'is[br]', 'the', 'fish[br]', 'no', 'really'] What I would like is to replace [br] with some fantastic value similar to
 and thus getting a new list: words = ['how', 'much', 'is
', 'the', 'fish
', 'no', 'really']

Output: words = [w.replace('[br]', '<br />') for w in words] This is called a list comprehension.


Instruction:
Get fully qualified class name of an object in Python

For logging purposes I want to retrieve the fully qualified class name of a Python object. (With fully qualified I mean the class name including the package and module name.) I know about x.__class__.__name__, but is there a simple method to get the package and module?

Output: With the following program #!/usr/bin/env python import foo def fullname(o): klass = o.__class__ module = klass.__module__ if module == 'builtins': return klass.__qualname__ # avoid outputs like 'builtins.str' return module + '.' + klass.__qualname__ bar = foo.Bar() print(fullname(bar)) and Bar defined as class Bar(object): def __init__(self, v=42): self.val = v the output is $ ./prog.py foo.Bar If you're still stuck on Python 2, you'll have to use __name__ instead of __qualname__, which is less informative for nested classes - a class Bar nested in a class Foo will show up as Bar instead of Foo.Bar: def fullname(o): klass = o.__class__ module = klass.__module__ if module == '__builtin__': return klass.__name__ # avoid outputs like '__builtin__.str' return module + '.' + klass.__name__


Instruction:
glob exclude pattern

I have a directory with a bunch of files inside: eee2314, asd3442 ... and eph. I want to exclude all files that start with eph with the glob function. How can I do it?

Output: The pattern rules for glob are not regular expressions. Instead, they follow standard Unix path expansion rules. There are only a few special characters: two different wild-cards, and character ranges are supported [from pymotw: glob – Filename pattern matching]. So you can exclude some files with patterns. For example to exclude manifests files (files starting with _) with glob, you can use: files = glob.glob('files_path/[!_]*')


Instruction:
Return first N key:value pairs from dict

Consider the following dictionary, d: d = {'a': 3, 'b': 2, 'c': 3, 'd': 4, 'e': 5} I want to return the first N key:value pairs from d (N <= 4 in this case). What is the most efficient method of doing this?

Output: Prior to Python 3.6 there is no such thing a the "first n" keys because a dict doesn't remember which keys were inserted first. You can get first n iteration sorted key-value pairs though: n_items = take(n, d.items()) This uses the implementation of take from the itertools recipes: from itertools import islice def take(n, iterable): """Return the first n items of the iterable as a list.""" return list(islice(iterable, n)) See it working online: ideone For Python < 3.6 n_items = take(n, d.iteritems())


Instruction:
Where do I call the BatchNormalization function in Keras?

If I want to use the BatchNormalization function in Keras, then do I need to call it once only at the beginning? I read this documentation for it: http://keras.io/layers/normalization/ I don't see where I'm supposed to call it. Below is my code attempting to use it: model = Sequential() keras.layers.normalization.BatchNormalization(epsilon=1e-06, mode=0, momentum=0.9, weights=None) model.add(Dense(64, input_dim=14, init='uniform')) model.add(Activation('tanh')) model.add(Dropout(0.5)) model.add(Dense(64, init='uniform')) model.add(Activation('tanh')) model.add(Dropout(0.5)) model.add(Dense(2, init='uniform')) model.add(Activation('softmax')) sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True) model.compile(loss='binary_crossentropy', optimizer=sgd) model.fit(X_train, y_train, nb_epoch=20, batch_size=16, show_accuracy=True, validation_split=0.2, verbose = 2) I ask because if I run the code with the second line including the batch normalization and if I run the code without the second line I get similar outputs. So either I'm not calling the function in the right place, or I guess it doesn't make that much of a difference.

Output: As Pavel said, Batch Normalization is just another layer, so you can use it as such to create your desired network architecture. The general use case is to use BN between the linear and non-linear layers in your network, because it normalizes the input to your activation function, so that you're centered in the linear section of the activation function (such as Sigmoid). There's a small discussion of it here In your case above, this might look like: # import BatchNormalization from keras.layers.normalization import BatchNormalization # instantiate model model = Sequential() # we can think of this chunk as the input layer model.add(Dense(64, input_dim=14, init='uniform')) model.add(BatchNormalization()) model.add(Activation('tanh')) model.add(Dropout(0.5)) # we can think of this chunk as the hidden layer model.add(Dense(64, init='uniform')) model.add(BatchNormalization()) model.add(Activation('tanh')) model.add(Dropout(0.5)) # we can think of this chunk as the output layer model.add(Dense(2, init='uniform')) model.add(BatchNormalization()) model.add(Activation('softmax')) # setting up the optimization of our weights sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True) model.compile(loss='binary_crossentropy', optimizer=sgd) # running the fitting model.fit(X_train, y_train, nb_epoch=20, batch_size=16, show_accuracy=True, validation_split=0.2, verbose = 2)


Instruction:
Matplotlib: draw grid lines behind other graph elements

In Matplotlib, I make dashed grid lines as follows: fig = pylab.figure() ax = fig.add_subplot(1,1,1) ax.yaxis.grid(color='gray', linestyle='dashed') however, I can't find out how (or even if it is possible) to make the grid lines be drawn behind other graph elements, such as bars. Changing the order of adding the grid versus adding other elements makes no difference. Is it possible to make it so that the grid lines appear behind everything else?

Output: According to this - https://web.archive.org/web/20200131000410/http://matplotlib.1069221.n5.nabble.com/axis-elements-and-zorder-td5346.html - you can use Axis.set_axisbelow(True) (I am currently installing matplotlib for the first time, so have no idea if that's correct - I just found it by googling "matplotlib z order grid" - "z order" is typically used to describe this kind of thing (z being the axis "out of the page"))


Instruction:
How can I get the version defined in setup.py (setuptools) in my package?

How could I get the version defined in setup.py from my package (for --version, or other purposes)?

Output: Interrogate version string of already-installed distribution To retrieve the version from inside your package at runtime (what your question appears to actually be asking), you can use: import pkg_resources # part of setuptools version = pkg_resources.require("MyProject")[0].version Store version string for use during install If you want to go the other way 'round (which appears to be what other answer authors here appear to have thought you were asking), put the version string in a separate file and read that file's contents in setup.py. You could make a version.py in your package with a __version__ line, then read it from setup.py using execfile('mypackage/version.py'), so that it sets __version__ in the setup.py namespace. Warning about race condition during install By the way, DO NOT import your package from your setup.py as suggested in another answer here: it will seem to work for you (because you already have your package's dependencies installed), but it will wreak havoc upon new users of your package, as they will not be able to install your package without manually installing the dependencies first.


Instruction:
How to plot a high resolution graph

I've used matplotlib for plotting some experimental results (discussed it in here: Looping over files and plotting. However, saving the picture by clicking right to the image gives very bad quality / low resolution images. from glob import glob import numpy as np import matplotlib.pyplot as plt import matplotlib as mpl # loop over all files in the current directory ending with .txt for fname in glob("./*.txt"): # read file, skip header (1 line) and unpack into 3 variables WL, ABS, T = np.genfromtxt(fname, skip_header=1, unpack=True) # first plot plt.plot(WL, T, label='BN', color='blue') plt.xlabel('Wavelength (nm)') plt.xlim(200,1000) plt.ylim(0,100) plt.ylabel('Transmittance, %') mpl.rcParams.update({'font.size': 14}) #plt.legend(loc='lower center') plt.title('') plt.show() plt.clf() # second plot plt.plot(WL, ABS, label='BN', color='red') plt.xlabel('Wavelength (nm)') plt.xlim(200,1000) plt.ylabel('Absorbance, A') mpl.rcParams.update({'font.size': 14}) #plt.legend() plt.title('') plt.show() plt.clf() Example graph of what I'm looking for: example graph

Output: You can use savefig() to export to an image file: plt.savefig('filename.png') In addition, you can specify the dpi argument to some scalar value (default is 100). For example: plt.savefig('filename.png', dpi=300)


Instruction:
Check if value already exists within list of dictionaries in Python?

I've got a Python list of dictionaries: a = [ {'main_color': 'red', 'second_color':'blue'}, {'main_color': 'yellow', 'second_color':'green'}, {'main_color': 'yellow', 'second_color':'blue'}, ] I'd like to check whether a dictionary with a particular key/value already exists in the list: // is a dict with 'main_color'='red' in the list already? // if not: add item

Output: if not any(d['main_color'] == 'red' for d in a): # does not exist The part in parentheses is a generator expression that returns True for each dictionary that has the key-value pair you are looking for, otherwise False. If the key could also be missing the above code can give you a KeyError. You can fix this by using get and providing a default value. If you don't provide a default value, None is returned. if not any(d.get('main_color', default_value) == 'red' for d in a): # does not exist


Instruction:
What is the EAFP principle in Python?

What is meant by "using the EAFP principle" in Python? Could you provide any examples?

Output: From the glossary: Easier to ask for forgiveness than permission. This common Python coding style assumes the existence of valid keys or attributes and catches exceptions if the assumption proves false. This clean and fast style is characterized by the presence of many try and except statements. The technique contrasts with the LBYL style common to many other languages such as C. An example would be an attempt to access a dictionary key. EAFP: try: x = my_dict["key"] except KeyError: # handle missing key LBYL: if "key" in my_dict: x = my_dict["key"] else: # handle missing key The LBYL version has to search the key inside the dictionary twice, and might also be considered slightly less readable.


Instruction:
open read and close a file in 1 line of code

Now I use: pageHeadSectionFile = open('pagehead.section.htm','r') output = pageHeadSectionFile.read() pageHeadSectionFile.close() But to make the code look better, I can do: output = open('pagehead.section.htm','r').read() When using the above syntax, how do I close the file to free up system resources?

Output: You don't really have to close it - Python will do it automatically either during garbage collection or at program exit. But as @delnan noted, it's better practice to explicitly close it for various reasons. So, what you can do to keep it short, simple and explicit: with open('pagehead.section.htm', 'r') as f: output = f.read() Now it's just two lines and pretty readable, I think.


Instruction:
How do you create nested dict in Python?

I have 2 CSV files: 'Data' and 'Mapping': 'Mapping' file has 4 columns: Device_Name, GDN, Device_Type, and Device_OS. All four columns are populated. 'Data' file has these same columns, with Device_Name column populated and the other three columns blank. I want my Python code to open both files and for each Device_Name in the Data file, map its GDN, Device_Type, and Device_OS value from the Mapping file. I know how to use dict when only 2 columns are present (1 is needed to be mapped) but I don't know how to accomplish this when 3 columns need to be mapped. Following is the code using which I tried to accomplish mapping of Device_Type: x = dict([]) with open("Pricing Mapping_2013-04-22.csv", "rb") as in_file1: file_map = csv.reader(in_file1, delimiter=',') for row in file_map: typemap = [row[0],row[2]] x.append(typemap) with open("Pricing_Updated_Cleaned.csv", "rb") as in_file2, open("Data Scraper_GDN.csv", "wb") as out_file: writer = csv.writer(out_file, delimiter=',') for row in csv.reader(in_file2, delimiter=','): try: row[27] = x[row[11]] except KeyError: row[27] = "" writer.writerow(row) It returns Attribute Error. After some researching, I think I need to create a nested dict, but I don't have any idea how to do this.

Output: A nested dict is a dictionary within a dictionary. A very simple thing. >>> d = {} >>> d['dict1'] = {} >>> d['dict1']['innerkey'] = 'value' >>> d['dict1']['innerkey2'] = 'value2' >>> d {'dict1': {'innerkey': 'value', 'innerkey2': 'value2'}} You can also use a defaultdict from the collections package to facilitate creating nested dictionaries. >>> import collections >>> d = collections.defaultdict(dict) >>> d['dict1']['innerkey'] = 'value' >>> d # currently a defaultdict type defaultdict(<type 'dict'>, {'dict1': {'innerkey': 'value'}}) >>> dict(d) # but is exactly like a normal dictionary. {'dict1': {'innerkey': 'value'}} You can populate that however you want. I would recommend in your code something like the following: d = {} # can use defaultdict(dict) instead for row in file_map: # derive row key from something # when using defaultdict, we can skip the next step creating a dictionary on row_key d[row_key] = {} for idx, col in enumerate(row): d[row_key][idx] = col According to your comment: may be above code is confusing the question. My problem in nutshell: I have 2 files a.csv b.csv, a.csv has 4 columns i j k l, b.csv also has these columns. i is kind of key columns for these csvs'. j k l column is empty in a.csv but populated in b.csv. I want to map values of j k l columns using 'i` as key column from b.csv to a.csv file My suggestion would be something like this (without using defaultdict): a_file = "path/to/a.csv" b_file = "path/to/b.csv" # read from file a.csv with open(a_file) as f: # skip headers f.next() # get first colum as keys keys = (line.split(',')[0] for line in f) # create empty dictionary: d = {} # read from file b.csv with open(b_file) as f: # gather headers except first key header headers = f.next().split(',')[1:] # iterate lines for line in f: # gather the colums cols = line.strip().split(',') # check to make sure this key should be mapped. if cols[0] not in keys: continue # add key to dict d[cols[0]] = dict( # inner keys are the header names, values are columns (headers[idx], v) for idx, v in enumerate(cols[1:])) Please note though, that for parsing csv files there is a csv module.


Instruction:
`staticmethod` and `abc.abstractmethod`: Will it blend?

In my Python app I want to make a method that is both a staticmethod and an abc.abstractmethod. How do I do this? I tried applying both decorators, but it doesn't work. If I do this: import abc class C(object): __metaclass__ = abc.ABCMeta @abc.abstractmethod @staticmethod def my_function(): pass I get an exception*, and if I do this: class C(object): __metaclass__ = abc.ABCMeta @staticmethod @abc.abstractmethod def my_function(): pass The abstract method is not enforced. How can I make an abstract static method? *The exception: File "c:\Python26\Lib\abc.py", line 29, in abstractmethod funcobj.__isabstractmethod__ = True AttributeError: 'staticmethod' object has no attribute '__isabstractmethod__'

Output: Starting with Python 3.3, it is possible to combine @staticmethod and @abstractmethod, so none of the other suggestions are necessary anymore: @staticmethod @abstractmethod def my_abstract_staticmethod(...): @abstractstaticmethod has been deprecated since version 3.3 (but is still there in Python 3.14).