Instruction:
How to use Jupyter notebooks in a conda environment?

Typically one runs jupyter notebook or jupyter-notebook or ipython notebook in a terminal to start a Jupyter notebook webserver locally (and open the URL in the browser). When using conda and conda environments, what is the best way to run a Jupyter notebook which allows to import Python modules installed in the conda environment? As it seems, this is not quite straight forward and many users have similar troubles. Most common error message seems to be: after installing a package XYZ in a conda environment my-env one can run import XYZ in a python console started in my-env, but running the same code in the Jupyter notebook will lead to an ImportError. This question has been asked many times, but there is no good place to answer it, most Q&A's and Github tickets are quite messy so let's start a new Q&A here.

Output: Disclaimer: ATM tested only in Ubuntu and Windows (see comments to this answer). Jupyter runs the user's code in a separate process called kernel. The kernel can be a different Python installation (in a different conda environment or virtualenv or Python 2 instead of Python 3) or even an interpreter for a different language (e.g. Julia or R). Kernels are configured by specifying the interpreter and a name and some other parameters (see Jupyter documentation) and configuration can be stored system-wide, for the active environment (or virtualenv) or per user. If nb_conda_kernels is used, additional to statically configured kernels, a separate kernel for each conda environment with ipykernel installed will be available in Jupyter notebooks. In short, there are three options how to use a conda environment and Jupyter: Option 1: Run Jupyter server and kernel inside the conda environment Do something like: conda create -n my-conda-env # creates new virtual env conda activate my-conda-env # activate environment in terminal conda install jupyter # install jupyter + notebook jupyter notebook # start server + kernel inside my-conda-env Jupyter will be completely installed in the conda environment. Different versions of Jupyter can be used for different conda environments, but this option might be a bit of overkill. It is enough to include the kernel in the environment, which is the component wrapping Python which runs the code. The rest of Jupyter notebook can be considered as editor or viewer and it is not necessary to install this separately for every environment and include it in every env.yml file. Therefore one of the next two options might be preferable, but this one is the simplest one and definitely fine. Option 2: Create special kernel for the conda environment Do something like: conda create -n my-conda-env # creates new virtual env conda activate my-conda-env # activate environment in terminal conda install ipykernel # install Python kernel in new conda env ipython kernel install --user --name=my-conda-env-kernel # configure Jupyter to use Python kernel Then run jupyter from the system installation or a different conda environment: conda deactivate # this step can be omitted by using a different terminal window than before conda install jupyter # optional, might be installed already in system e.g. by 'apt install jupyter' on debian-based systems jupyter notebook # run jupyter from system Name of the kernel and the conda environment are independent from each other, but it might make sense to use a similar name. Only the Python kernel will be run inside the conda environment, Jupyter from system or a different conda environment will be used - it is not installed in the conda environment. By calling ipython kernel install the jupyter is configured to use the conda environment as kernel, see Jupyter documentation and IPython documentation for more information. In most Linux installations this configuration is a *.json file in ~/.local/share/jupyter/kernels/my-conda-env-kernel/kernel.json: { "argv": [ "/opt/miniconda3/envs/my-conda-env/bin/python", "-m", "ipykernel_launcher", "-f", "{connection_file}" ], "display_name": "my-conda-env-kernel", "language": "python" } Option 3: Use nb_conda_kernels to use a kernel in the conda environment When the package nb_conda_kernels is installed, a separate kernel is available automatically for each conda environment containing the conda package ipykernel or a different kernel (R, Julia, ...). conda activate my-conda-env # this is the environment for your project and code conda install ipykernel conda deactivate conda activate base # could be also some other environment conda install nb_conda_kernels jupyter notebook You should be able to choose the Kernel Python [conda env:my-conda-env]. Note that nb_conda_kernels seems to be available only via conda and not via pip or other package managers like apt. Troubleshooting Using Linux/Mac the command which on the command line will tell you which jupyter is used, if you are using option 1 (running Jupyter from inside the conda environment), it should be an executable from your conda environment: $ which jupyter /opt/miniconda3/envs/my-conda-env/bin/jupyter $ which jupyter-notebook # this might be different than 'which jupyter'! (see below) /opt/miniconda3/envs/my-conda-env/bin/jupyter-notebook Inside the notebook you should see that Python uses Python paths from the conda environment: [1] !which python /opt/miniconda3/envs/my-conda-env/bin/python [2] import sys; sys.executable '/opt/miniconda3/envs/my-conda-env/bin/python' ['/home/my_user', '/opt/miniconda3/envs/my-conda-env/lib/python37.zip', '/opt/miniconda3/envs/my-conda-env/lib/python3.7', '/opt/miniconda3/envs/my-conda-env/lib/python3.7/lib-dynload', '', '/opt/miniconda3/envs/my-conda-env/lib/python3.7/site-packages', '/opt/miniconda3/envs/my-conda-env/lib/python3.7/site-packages/IPython/extensions', '/home/my_user/.ipython'] Jupyter provides the command jupyter-troubleshoot or in a Jupyter notebook: !jupyter-troubleshoot This will print a lot of helpful information about including the outputs mentioned above as well as installed libraries and others. When asking for help regarding Jupyter installations questions, it might be good idea to provide this information in bug reports or questions. To list all configured Jupyter kernels run: jupyter kernelspec list Common errors and traps Jupyter notebook not installed in conda environment Note: symptoms are not unique to the issue described here. Symptoms: ImportError in Jupyter notebooks for modules installed in the conda environment (but not installed system wide), but no error when importing in a Python terminal Explaination: You tried to run jupyter notebook from inside your conda environment (option 1, see above), there is no configuration for a kernel for this conda environment (this would be option 2) and nb_conda_kernels is not installed (option 3), but jupyter notebook is not (fully) installed in the conda environment, even if which jupyter might make you believe it was. In GNU/Linux you can type which jupyter to check which executable of Jupyter is run. This means that system's Jupyter is used, probably because Jupyter is not installed: (my-conda-env) $ which jupyter-notebook /usr/bin/jupyter If the path points to a file in your conda environment, Jupyter is run from inside Jupyter: (my-conda-env) $ which jupyter-notebook /opt/miniconda3/envs/my-conda-env/bin/jupyter-notebook Note that when the conda package ipykernel is installed, an executable jupyter is shipped, but no executable jupyter-notebook. This means that which jupyter will return a path to the conda environment but jupyter notebook will start system's jupyter-nootebook (see also here): $ conda create -n my-conda-env $ conda activate my-conda-env $ conda install ipykernel $ which jupyter # this looks good, but is misleading! /opt/miniconda3/envs/my-conda-env/bin/jupyter $ which jupyter-notebook # jupyter simply runs jupyter-notebook from system... /usr/bin/jupyter-notebook This happens because jupyter notebook searches for jupyter-notebook, finds /usr/bin/jupyter-notebook and calls it starting a new Python process. The shebang in /usr/bin/jupyter-notebook is #!/usr/bin/python3 and not a dynamic #!/usr/bin/env python. Therefore Python manages to break out of the conda environment. I guess jupyter could call python /usr/bin/jupyter-notebook instead to overrule the shebang, but mixing system's bin files and the environment's python path can't work well anyway. Solution: Install jupyter notebook inside the conda environment: conda activate my-conda-env conda install jupyter jupyter notebook Wrong kernel configuration: Kernel is configured to use system Python Note: symptoms are not unique to the issue described here. Symptoms: ImportError in Jupyter notebooks for modules installed in the conda environment (but not installed system wide), but no error when importing in a Python terminal Explanation: Typically the system provides a kernel called python3 (display name "Python 3") configured to use /usr/bin/python3, see e.g. /usr/share/jupyter/kernels/python3/kernel.json. This is usually overridden by a kernel in the conda environment, which points to the environments python binary /opt/miniconda3/envs/my-conda-env/bin/python. Both are generated by the package ipykernel (see here and here). A user kernel specification in ~/.local/share/jupyter/kernels/python3/kernel.json might override the system-wide and environment kernel. If the environment kernel is missing or the user kernel points to a python installation outside the environment option 1 (installation of jupyter in the environment) will fail. For occurrences and discussions of this problem and variants see here, here, here and also here, here and here. Solution: Use jupyter kernelspec list to list the location active kernel locations. $ conda activate my-conda-env $ jupyter kernelspec list Available kernels: python3 /opt/miniconda3/envs/my-conda-env/share/jupyter/kernels/python3 If the kernel in the environment is missing, you can try creating it manually using ipython kernel install --sys-prefix in the activated environment, but it is probably better to check your installation, because conda install ipykernel should have created the environment (maybe try re-crate the environment and re-install all packages?). If a user kernel specification is blocking the environment kernel specification, you can either remove it or use a relative python path which will use $PATH to figure out which python to use. So something like this, should be totally fine: $ cat ~/.local/share/jupyter/kernels/python3/kernel.json { "argv": [ "python", "-m", "ipykernel_launcher", "-f", "{connection_file}" ], "display_name": "Python 3", "language": "python" } Correct conda environment not activated Symptoms: ImportError for modules installed in the conda environment (but not installed system wide) in Jupyter notebooks and Python terminals Explanation: Each terminal has a set of environment variables, which are lost when the terminal is closed. In order to use a conda environment certain environment variables need to be set, which is done by activating it using conda activate my-conda-env. If you attempted to run Jupyter notebook from inside the conda environment (option 1), but did not activate the conda environment before running it, it might run the system's jupyter. Solution: Activate conda environment before running Jupyter. conda activate my-conda-env jupyter notebook Broken kernel configuration Symptoms: Strange things happening. Maybe similar symptoms as above, e.g. ImportError Explanation: If you attempted to use option 2, i.e. running Jupyter from system and the Jupyter kernel inside the conda environment by using an explicit configuration for the kernel, but it does not behave as you expect, the configuration might be corrupted in some way. Solution: Check configuration in ~/.local/share/jupyter/kernels/my-kernel-name/kernel.json and fix mistakes manually or remove the entire directory and re-create it using the command provided above for option 2. If you can't find the kernel configuration there run jupyter kernelspec list. Python 2 vs 3 Symptoms: ImportError due to wrong Python version of the Jupyter kernel or other problems with Python 2/3 Explanation: The kernel configuration can have all sorts of confusing and misleading effects. For example the default Python 3 kernel configuration will allow me to launch a Jupyter notebook running on Python 2: conda create -n my-conda-env conda activate my-conda-env conda install python=2 conda install jupyter jupyter notebook The default Python 3 kernel: $ cat ~/.local/share/jupyter/kernels/python3/kernel.json { "argv": [ "python", "-m", "ipykernel_launcher", "-f", "{connection_file}" ], "display_name": "Python 3", "language": "python" } After creating a new Jupyter Notebook with the Python 3 kernel, Python 2 from the conda environment will be used even if "Python 3" is displayed by Jupyter. Solution: Don't use Python 2 ;-)


Instruction:
Python Pandas equivalent in JavaScript

With this CSV example: Source,col1,col2,col3 foo,1,2,3 bar,3,4,5 The standard method I use Pandas is this: Parse CSV Select columns into a data frame (col1 and col3) Process the column (e.g. avarage the values of col1 and col3) Is there a JavaScript library that does that like Pandas?

Output: This wiki will summarize and compare many pandas-like Javascript libraries. In general, you should check out the d3 JavaScript library. d3 is very useful "swiss army knife" for handling data in JavaScript, just like pandas is helpful for Python. You may see d3 used frequently like pandas, even if d3 is not exactly a DataFrame/Pandas replacement (i.e. d3 doesn't have the same API; d3 does not have Series / DataFrame classes with methods that match the pandas behavior). Ahmed's answer explains how d3 can be used to achieve some DataFrame functionality, and some of the libraries below were inspired by things like LearnJsData which uses d3 and lodash. As for DataFrame-style data transformation (splitting, joining, group by etc), here is a quick list of some of the JavaScript libraries. Note the libraries are written in different languages, including... browser-compatible aka client-side JavaScript Node.js aka Server-side JavaScript Typescript Some even use CPython transpiled to WebAssembly (but work with Node.js and/or browsers) ...so use the option that's right for you: Pyodide (browser-support AND Nodejs-support) A new but very strong contender from Ahmed Fasih's answer WebAssembly transpilation of CPython and a large portion of the numeric Python ecosystem ("including NumPy, pandas, SciPy, Matplotlib, and scikit-learn") "for the browser and Node.js " made by folks tightly connected to Project Jupyter see its list of included packages danfo-js (browser-support AND NodeJS-support) From Vignesh's answer danfo (which is often imported and aliased as dfd); has a basic DataFrame-type data structure, with the ability to plot directly Built by the team at Tensorflow: "One of the main goals of Danfo.js is to bring data processing, machine learning and AI tools to JavaScript developers. ... Open-source libraries like Numpy and Pandas..." pandas is built on top of numpy; likewise danfo-js is built on tensorflow-js please note danfo may not (yet?) support multi-column indexes pandas-js UPDATE The pandas-js repo has not been updated in awhile From STEEL and Feras' answers "pandas.js is an open source (experimental) library mimicking the Python pandas library. It relies on Immutable.js as the NumPy logical equivalent. The main data objects in pandas.js are, like in Python pandas, the Series and the DataFrame." dataframe-js "DataFrame-js provides an immutable data structure for javascript and datascience, the DataFrame, which allows to work on rows and columns with a sql and functional programming inspired api." data-forge Seen in Ashley Davis' answer "JavaScript data transformation and analysis toolkit inspired by Pandas and LINQ." Note the old data-forge JS repository is no longer maintained; now a new repository uses Typescript jsdataframe "Jsdataframe is a JavaScript data wrangling library inspired by data frame functionality in R and Python Pandas." dataframe "explore data by grouping and reducing." SQL Frames "DataFrames meet SQL, in the Browser" "SQL Frames is a low code data management framework that can be directly embedded in the browser to provide rich data visualization and UX. Complex DataFrames can be composed using familiar SQL constructs. With its powerful built-in analytics engine, data sources can come in any shape, form and frequency and they can be analyzed directly within the browser. It allows scaling to big data backends by transpiling the composed DataFrame logic to SQL." Jandas (browser- AND NodeJS-support; a new TypeScript library developed in 2023) Indexing and query very similar to Pandas Negative number and range indexing Support DataFrame with zero rows/columns Support index with duplicated values iterrows, sort_values, groupby and element-wise operation Then after coming to this question, checking other answers here and doing more searching, I found options like: Apache Arrow in JS Thanks to user Back2Basics suggestion: "Apache Arrow is a columnar memory layout specification for encoding vectors and table-like containers of flat and nested data. Apache Arrow is the emerging standard for large in-memory columnar data (Spark, Pandas, Drill, Graphistry, ...)" polars Polars is a blazingly fast DataFrames library implemented in Rust using Apache Arrow Columnar Format as memory model. Observable At first glance, seems like a JS alternative to the IPython/Jupyter "notebooks" Observable's page promises: "Reactive programming", a "Community", on a "Web Platform" See 5 minute intro here portal.js (formerly recline; from Rufus' answer) MAY BE OUTDATED: Does not use a "DataFrame" API MAY BE OUTDATED: Instead emphasizes its "Multiview" (the UI) API, (similar to jQuery/DOM model) which doesn't require jQuery but does require a browser! More examples MAY BE OUTDATED: Also emphasizes its MVC-ish architecture; including back-end stuff (i.e. database connections) js-data Really more of an ORM! Most of its modules correspond to different data storage questions (js-data-mongodb, js-data-redis, js-data-cloud-datastore), sorting, filtering, etc. On plus-side does work on Node.js as a first-priority; "Works in Node.js and in the Browser." miso (another suggestion from Rufus) Impressive backers like Guardian and bocoup. AlaSQL "AlaSQL" is an open source SQL database for Javascript with a strong focus on query speed and data source flexibility for both relational data and schemaless data. It works in your browser, Node.js, and Cordova." Some thought experiments: "Scaling a DataFrame in Javascript" - Gary Sieling Here are the criteria we used to consider the above choices General Criteria Language (NodeJS vs browser JS vs Typescript) Dependencies (i.e. if it uses an underlying library / AJAX/remote API's) Actively supported (active user-base, active source repository, etc) Size/speed of JS library Panda's criterias in its R comparison Performance Functionality/flexibility Ease-of-use Similarity to Pandas / Dataframe API's Specifically hits on their main features Data-science emphasis Built-in visualization functions Demonstrated integration in combination with other tools like Jupyter (interactive notebooks), etc


Instruction:
"gettext()" vs "gettext_lazy()" in Django

I have a question about using ugettext and gettext_lazy() for translations. I learned that in models I should use gettext_lazy(), while in views ugettext. But are there any other places, where I should use gettext_lazy() too? What about form definitions? Are there any performance diffrences between them? Edit: And one more thing. Sometimes, instead of gettext_lazy(), gettext_noop() is used. As documentation says, gettext_noop() strings are only marked for translation and translated at the latest possible momment before displaying them to the user, but I'm little confused here, isn't that similar to what gettext_lazy() do? It's still hard for me to decide, which should I use in my models and forms.

Output: gettext() vs. gettext_lazy() In definitions like forms or models you should use gettext_lazy because the code of this definitions is only executed once (mostly on django's startup); gettext_lazy translates the strings in a lazy fashion, which means, eg. every time you access the name of an attribute on a model the string will be newly translated-which totally makes sense because you might be looking at this model in different languages since django was started! In views and similar function calls you can use gettext without problems, because everytime the view is called gettext will be newly executed, so you will always get the right translation fitting the request! Regarding gettext_noop() As Bryce pointed out in his answer, this function marks a string as extractable for translation but does return the untranslated string. This is useful for using the string in two places – translated and untranslated. See the following example: import logging from django.http import HttpResponse from django.utils.translation import gettext as _, gettext_noop as _noop def view(request): msg = _noop("An error has occurred") logging.error(msg) return HttpResponse(_(msg))


Instruction:
"Private" (implementation) class in Python

I am coding a small Python module composed of two parts: some functions defining a public interface, an implementation class used by the above functions, but which is not meaningful outside the module. At first, I decided to "hide" this implementation class by defining it inside the function using it, but this hampers readability and cannot be used if multiple functions reuse the same class. So, in addition to comments and docstrings, is there a mechanism to mark a class as "private" or "internal"? I am aware of the underscore mechanism, but as I understand it it only applies to variables, function and methods name.

Output: Use a single underscore prefix: class _Internal: ... This is the official Python convention for 'internal' symbols; from module import * does not import underscore-prefixed objects. Reference to the single-underscore convention in the Python 2 and Python 3 documentation.


Instruction:
How do you do natural logs (e.g. "ln()") with numpy in Python?

Using numpy, how can I do the following: ln(x) Is it equivalent to: np.log(x) I apologise for such a seemingly trivial question, but my understanding of the difference between log and ln is that ln is logspace e?

Output: np.log is ln, whereas np.log10 is your standard base 10 log.


Instruction:
What are good uses for Python3's "Function Annotations"?

Function Annotations: PEP-3107 I ran across a snippet of code demonstrating Python3's function annotations. The concept is simple but I can't think of why these were implemented in Python3 or any good uses for them. Perhaps SO can enlighten me? How it works: def foo(a: 'x', b: 5 + 6, c: list) -> max(2, 9): ... function body ... Everything following the colon after an argument is an 'annotation', and the information following the -> is an annotation for the function's return value. foo.func_annotations would return a dictionary: {'a': 'x', 'b': 11, 'c': list, 'return': 9} What's the significance of having this available?

Output: I think this is actually great. Coming from an academic background, I can tell you that annotations have proved themselves invaluable for enabling smart static analyzers for languages like Java. For instance, you could define semantics like state restrictions, threads that are allowed to access, architecture limitations, etc., and there are quite a few tools that can then read these and process them to provide assurances beyond what you get from the compilers. You could even write things that check preconditions/postconditions. I feel something like this is especially needed in Python because of its weaker typing, but there were really no constructs that made this straightforward and part of the official syntax. There are other uses for annotations beyond assurance. I can see how I could apply my Java-based tools to Python. For instance, I have a tool that lets you assign special warnings to methods, and gives you indications when you call them that you should read their documentation (E.g., imagine you have a method that must not be invoked with a negative value, but it's not intuitive from the name). With annotations, I could technically write something like this for Python. Similarly, a tool that organizes methods in a large class based on tags can be written if there is an official syntax.


Instruction:
Python: Is it bad form to raise exceptions within __init__?

Is it considered bad form to raise exceptions within __init__? If so, then what is the accepted method of throwing an error when certain class variables are initialized as None or of an incorrect type?

Output: Raising exceptions within __init__() is absolutely fine. There's no other good way to indicate an error condition within an initializer, and there are many hundreds of examples in the standard library where initializing an object can raise an exception. The error class to raise, of course, is up to you. ValueError is best if the initializer was passed an invalid parameter.


Instruction:
redis-py : What's the difference between StrictRedis() and Redis()?

I want to use redis-py for caching some data, but I can't find a suitable explanation of the difference between redis.StrictRedis() and redis.Redis(). Are they equivalent? In addition, I can't find any clear documentation about redis.StrictRedis()'s arguments in Redis Python Docs. Any idea?

Output: EDIT: They are now equivalent: redis-py 3.0 drops support for the legacy "Redis" client class. "StrictRedis" has been renamed to "Redis" and an alias named "StrictRedis" is provided so that users previously using "StrictRedis" can continue to run unchanged. Original answer: This seems pretty clear: redis-py exposes two client classes that implement these commands The StrictRedis class attempts to adhere to the official command syntax. and In addition to the changes above, the Redis class, a subclass of StrictRedis, overrides several other commands to provide backwards compatibility with older versions of redis-py Do you need backwards compatibility? Use Redis. Don't care? Use StrictRedis. 2017-03-31 Here are the specifics of the backwards compatibility, from the github.com link cited: In addition to the changes above, the Redis class, a subclass of StrictRedis, overrides several other commands to provide backwards compatibility with older versions of redis-py: LREM: Order of 'num' and 'value' arguments reversed such that 'num' can provide a default value of zero. ZADD: Redis specifies the 'score' argument before 'value'. These were swapped accidentally when being implemented and not discovered until after people were already using it. The Redis class expects *args in the form of: name1, score1, name2, score2, ... SETEX: Order of 'time' and 'value' arguments reversed.


Instruction:
How to index into a dictionary?

I have a Dictionary below: colors = { "blue" : "5", "red" : "6", "yellow" : "8", } How do I index the first entry in the dictionary? colors[0] will return a KeyError for obvious reasons.

Output: Dictionaries are unordered in Python versions up to and including Python 3.6. If you do not care about the order of the entries and want to access the keys or values by index anyway, you can create a list of keys for a dictionary d using keys = list(d), and then access keys in the list by index keys[i], and the associated values with d[keys[i]]. If you do care about the order of the entries, starting with Python 2.7 you can use collections.OrderedDict. Or use a list of pairs l = [("blue", "5"), ("red", "6"), ("yellow", "8")] if you don't need access by key. (Why are your numbers strings by the way?) In Python 3.7, normal dictionaries are ordered, so you don't need to use OrderedDict anymore (but you still can – it's basically the same type). The CPython implementation of Python 3.6 already included that change, but since it's not part of the language specification, you can't rely on it in Python 3.6.


Instruction:
Replace console output in Python

I'm wondering how I could create one of those nifty console counters in Python as in certain C/C++-programs. I've got a loop doing things and the current output is along the lines of: Doing thing 0 Doing thing 1 Doing thing 2 ... what would be neater would be to just have the last line update; X things done. I've seen this in a number of console programs and am wondering if/how I'd do this in Python.

Output: An easy solution is just writing "\r" before the string and not adding a newline; if the string never gets shorter this is sufficient... sys.stdout.write("\rDoing thing %i" % i) sys.stdout.flush() Slightly more sophisticated is a progress bar... this is something I am using: def start_progress(title): global progress_x sys.stdout.write(title + ": [" + "-"*40 + "]" + chr(8)*41) sys.stdout.flush() progress_x = 0 def progress(x): global progress_x x = int(x * 40 // 100) sys.stdout.write("#" * (x - progress_x)) sys.stdout.flush() progress_x = x def end_progress(): sys.stdout.write("#" * (40 - progress_x) + "]\n") sys.stdout.flush() You call start_progress passing the description of the operation, then progress(x) where x is the percentage and finally end_progress()


Instruction:
How to generate a random number with a specific amount of digits?

Let's say I need a 3-digit number, so it would be something like: >>> random(3) 563 or >>> random(5) 26748 >> random(2) 56

Output: You can use either of random.randint or random.randrange. So to get a random 3-digit number: from random import randint, randrange randint(100, 999) # randint is inclusive at both ends randrange(100, 1000) # randrange is exclusive at the stop * Assuming you really meant three digits, rather than "up to three digits". To use an arbitrary number of digits: from random import randint def random_with_N_digits(n): range_start = 10**(n-1) range_end = (10**n)-1 return randint(range_start, range_end) print random_with_N_digits(2) print random_with_N_digits(3) print random_with_N_digits(4) Output: 33 124 5127


Instruction:
Putting text in top left corner of matplotlib plot

How can I put text in the top left (or top right) corner of a matplotlib figure, e.g. where a top left legend would be, or on top of the plot but in the top left corner? E.g. if it's a plt.scatter(), then something that would be within the square of the scatter, put in the top left most corner. I'd like to do this without ideally knowing the scale of the scatterplot being plotted for example, since it will change from dataset to data set. I just want it the text to be roughly in the upper left, or roughly in the upper right. With legend type positioning it should not overlap with any scatter plot points anyway.

Output: You can use text. plt.text(x, y, s, fontsize=12) text coordinates can be given relative to the axis, so the position of your text will be independent of the size of the plot: The default transform specifies that text is in data coords, alternatively, you can specify text in axis coords (0,0 is lower-left and 1,1 is upper-right). The example below places text in the center of the axes:: plt.text(0.5, 0.5, 'matplotlib', horizontalalignment='center', verticalalignment='center', transform = ax.transAxes) To prevent the text to interfere with any point of your scatter is more difficult afaik. The easier method is to set y_axis (ymax in ylim((ymin,ymax))) to a value a bit higher than the max y-coordinate of your points. In this way you will always have this free space for the text. EDIT: here you have an example: from matplotlib import pyplot as plt f, ax = plt.subplots() plt.scatter([3,5,2,6,8],[5,3,2,1,5]) plt.text(.01, .99, 'matplotlib', ha='left', va='top', transform=ax.transAxes) f.tight_layout() The ha and va parameters set the alignment of your text relative to the insertion point. ie. ha='left' is a good set to prevent a long text to go out of the left axis when the frame is reduced (made narrower) manually.


Instruction:
How to connect scatterplot points with line using matplotlib

I have two lists, dates and values. I want to plot them using matplotlib. The following creates a scatter plot of my data. import matplotlib.pyplot as plt plt.scatter(dates,values) plt.show() plt.plot(dates, values) creates a line graph. But what I really want is a scatterplot where the points are connected by a line. Similar to in R: plot(dates, values) lines(dates, value, type="l") which gives me a scatterplot of points overlaid with a line connecting the points. How do I do this in python?

Output: I think @Evert has the right answer: plt.scatter(dates,values) plt.plot(dates, values) plt.show() Which is pretty much the same as plt.plot(dates, values, '-o') plt.show() You can replace -o with another suitable format string as described in the documentation. You can also split the choices of line and marker styles using the linestyle= and marker= keyword arguments.


Instruction:
Abstract attributes in Python

What is the shortest / most elegant way to implement the following Scala code with an abstract attribute in Python? abstract class Controller { val path: String } A subclass of Controller is enforced to define "path" by the Scala compiler. A subclass would look like this: class MyController extends Controller { override val path = "/home" }

Output: Python has a built-in exception for this, though you won't encounter the exception until runtime. class Base(object): @property def path(self): raise NotImplementedError class SubClass(Base): path = 'blah'


Instruction:
Convert image from PIL to openCV format

I'm trying to convert image from PIL to OpenCV format. I'm using OpenCV 2.4.3. here is what I've attempted till now. >>> from PIL import Image >>> import cv2 as cv >>> pimg = Image.open('D:\\traffic.jpg') #PIL Image >>> cimg = cv.cv.CreateImageHeader(pimg.size,cv.IPL_DEPTH_8U,3) #CV Image >>> cv.cv.SetData(cimg,pimg.tostring()) >>> cv.cv.NamedWindow('cimg') >>> cv.cv.ShowImage('cimg',cimg) >>> cv.cv.WaitKey() But I think the image is not getting converted to CV format. The Window shows me a large brown image. Where am I going wrong in Converting image from PIL to CV format? Also, why do I need to type cv.cv to access functions?

Output: Use this: pil_image = PIL.Image.open('Image.jpg').convert('RGB') open_cv_image = numpy.array(pil_image) # Convert RGB to BGR open_cv_image = open_cv_image[:, :, ::-1].copy()


Instruction:
Django TemplateSyntaxError - 'staticfiles' is not a registered tag library

After upgrading to Django 3.0, I get the following TemplateSyntaxError: In template /Users/alasdair//myproject/myapp/templates/index.html, error at line 1 'staticfiles' is not a registered tag library. Must be one of: admin_list admin_modify admin_urls cache i18n l10n log static tz Here is my template {% load staticfiles %} <img src="{% static 'my_image.html' %}">

Output: If you have any of the following tags in your template: {% load staticfiles %} {% load static from staticfiles %} {% load admin_static %} Then replace it with: {% load static %} You have to make this change because {% load staticfiles %} and {% load admin_static %} were deprecated in Django 2.1, and removed in Django 3.0.


Instruction:
<Django object > is not JSON serializable

I have the following code for serializing the queryset: def render_to_response(self, context, **response_kwargs): return HttpResponse(json.simplejson.dumps(list(self.get_queryset())), mimetype="application/json") And following is my get_quersety() [{'product': <Product: hederello ()>, u'_id': u'9802', u'_source': {u'code': u'23981', u'facilities': [{u'facility': {u'name': {u'fr': u'G\xe9n\xe9ral', u'en': u'General'}, u'value': {u'fr': [u'bar', u'r\xe9ception ouverte 24h/24', u'chambres non-fumeurs', u'chambres familiales',.........]}] Which I need to serialize. But it says not able to serialize the <Product: hederello ()>. Because the list is composed of both django objects and dicts. Any ideas?

Output: simplejson and json don't work with django objects well. Django's built-in serializers can only serialize querysets filled with django objects: data = serializers.serialize('json', self.get_queryset()) return HttpResponse(data, content_type="application/json") In your case, self.get_queryset() contains a mix of django objects and dicts inside. One option is to get rid of model instances in the self.get_queryset() and replace them with dicts using model_to_dict: from django.forms.models import model_to_dict data = self.get_queryset() for item in data: item['product'] = model_to_dict(item['product']) return HttpResponse(json.simplejson.dumps(data), mimetype="application/json")


Instruction:
Calculate Time Difference Between Two Pandas Columns in Hours and Minutes

I have two columns, fromdate and todate, in a dataframe. import pandas as pd data = {'todate': [pd.Timestamp('2014-01-24 13:03:12.050000'), pd.Timestamp('2014-01-27 11:57:18.240000'), pd.Timestamp('2014-01-23 10:07:47.660000')], 'fromdate': [pd.Timestamp('2014-01-26 23:41:21.870000'), pd.Timestamp('2014-01-27 15:38:22.540000'), pd.Timestamp('2014-01-23 18:50:41.420000')]} df = pd.DataFrame(data) I add a new column, diff, to find the difference between the two dates using df['diff'] = df['fromdate'] - df['todate'] I get the diff column, but it contains days, when there's more than 24 hours. todate fromdate diff 0 2014-01-24 13:03:12.050 2014-01-26 23:41:21.870 2 days 10:38:09.820000 1 2014-01-27 11:57:18.240 2014-01-27 15:38:22.540 0 days 03:41:04.300000 2 2014-01-23 10:07:47.660 2014-01-23 18:50:41.420 0 days 08:42:53.760000 How do I convert my results to only hours and minutes (i.e. days are converted to hours)?

Output: Pandas timestamp differences returns a datetime.timedelta object. This can easily be converted into hours by using the as_type method, like so import pandas as pd df = pd.DataFrame(columns=['to', 'fr', 'ans']) df.to = [pd.Timestamp('2014-01-24 13:03:12.050000'), pd.Timestamp('2014-01-27 11:57:18.240000'), pd.Timestamp('2014-01-23 10:07:47.660000')] df.fr = [pd.Timestamp('2014-01-26 23:41:21.870000'), pd.Timestamp('2014-01-27 15:38:22.540000'), pd.Timestamp('2014-01-23 18:50:41.420000')] (df.fr - df.to).astype('timedelta64[h]') to yield, 0 58 1 3 2 8 dtype: float64


Instruction:
Unittest setUp/tearDown for several tests

Is there a function that is fired at the beginning/end of a scenario of tests? The functions setUp and tearDown are fired before/after every single test. I typically would like to have this: class TestSequenceFunctions(unittest.TestCase): def setUpScenario(self): start() #launched at the beginning, once def test_choice(self): element = random.choice(self.seq) self.assertTrue(element in self.seq) def test_sample(self): with self.assertRaises(ValueError): random.sample(self.seq, 20) for element in random.sample(self.seq, 5): self.assertTrue(element in self.seq) def tearDownScenario(self): end() #launched at the end, once For now, these setUp and tearDown are unit tests and spread in all my scenarios (containing many tests), one is the first test, the other is the last test.

Output: As of 2.7 (per the documentation) you get setUpClass and tearDownClass which execute before and after the tests in a given class are run, respectively. Alternatively, if you have a group of them in one file, you can use setUpModule and tearDownModule (documentation). EDIT: Note that setUpClass and tearDownClass must be declared using @classmethod. Another answer to this question includes sample code. Otherwise your best bet is probably going to be to create your own derived TestSuite and override run(). All other calls would be handled by the parent, and run would call your setup and teardown code around a call up to the parent's run method.


Instruction:
How to remove leading and trailing spaces from a string?

I'm having a hard time trying to use .strip with the following line of code: f.write(re.split("Tech ID:|Name:|Account #:",line)[-1])

Output: You can use the strip() method to remove trailing and leading spaces: >>> s = ' abd cde ' >>> s.strip() 'abd cde' Note: the internal spaces are preserved.


Instruction:
Set logging levels

I'm trying to use the standard library to debug my code: This works fine: import logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) logger.info('message') I can't make work the logger for the lower levels: logging.basicConfig(level=logging.DEBUG) logger = logging.getLogger(__name__) logger.info('message') logging.basicConfig(level=logging.DEBUG) logger = logging.getLogger(__name__) logger.debug('message') I don't get any response for neither of those.

Output: What Python version? That works for me in 3.4. But note that basicConfig() won't affect the root handler if it's already setup: This function does nothing if the root logger already has handlers configured for it. To set the level on root explicitly do logging.getLogger().setLevel(logging.DEBUG). But ensure you've called basicConfig() before hand so the root logger initially has some setup. I.e.: import logging logging.basicConfig() logging.getLogger().setLevel(logging.DEBUG) logging.getLogger('foo').debug('bah') logging.getLogger().setLevel(logging.INFO) logging.getLogger('foo').debug('bah') Also note that "Loggers" and their "Handlers" both have distinct independent log levels. So if you've previously explicitly loaded some complex logger config in you Python script, and that has messed with the root logger's handler(s), then this can have an effect, and just changing the loggers log level with logging.getLogger().setLevel(..) may not work. This is because the attached handler may have a log level set independently. This is unlikely to be the case and not something you'd normally have to worry about.


Instruction:
Understand Python swapping: why is a, b = b, a not always equivalent to b, a = a, b?

As we all know, the pythonic way to swap the values of two items a and b is a, b = b, a and it should be equivalent to b, a = a, b However, today when I was working on some code, I accidentally found that the following two swaps give different results: nums = [1, 2, 4, 3] i = 2 nums[i], nums[nums[i]-1] = nums[nums[i]-1], nums[i] print(nums) # [1, 2, 4, 3] nums = [1, 2, 4, 3] i = 2 nums[nums[i]-1], nums[i] = nums[i], nums[nums[i]-1] print(nums) # [1, 2, 3, 4] What is happening here? I thought in a Python swap the two assignments happen simultaneously and independently. See also Multiple assignment and evaluation order in Python regarding the basic semantics of this kind of assignment. See also Multiple assignment semantics regarding the effect and purpose of parentheses on the left-hand side of a multiple assignment.

Output: From python.org Assignment of an object to a target list, optionally enclosed in parentheses or square brackets, is recursively defined as follows. ... Else: The object must be an iterable with the same number of items as there are targets in the target list, and the items are assigned, from left to right, to the corresponding targets. So I interpret that to mean that your assignment nums[i], nums[nums[i]-1] = nums[nums[i]-1], nums[i] is roughly equivalent to tmp = nums[nums[i]-1], nums[i] nums[i] = tmp[0] nums[nums[i] - 1] = tmp[1] (with better error-checking, of course) whereas the other nums[nums[i]-1], nums[i] = nums[i], nums[nums[i]-1] is like tmp = nums[i], nums[nums[i]-1] nums[nums[i] - 1] = tmp[0] nums[i] = tmp[1] So the right-hand side is evaluated first in both cases. But then the two pieces of the left-hand side are evaluated in order, and the assignments are done immediately after evaluation. Crucially, this means that the second term on the left-hand side is only evaluated after the first assignment is already done. So if you update nums[i] first, then the nums[nums[i] - 1] refers to a different index than if you update nums[i] second.


Instruction:
How to upload a file to directory in S3 bucket using boto

I want to copy a file in s3 bucket using python. Ex : I have bucket name = test. And in the bucket, I have 2 folders name "dump" & "input". Now I want to copy a file from local directory to S3 "dump" folder using python... Can anyone help me?

Output: NOTE: This answer uses boto. See the other answer that uses boto3, which is newer. Try this... import boto import boto.s3 import sys from boto.s3.key import Key AWS_ACCESS_KEY_ID = '' AWS_SECRET_ACCESS_KEY = '' bucket_name = AWS_ACCESS_KEY_ID.lower() + '-dump' conn = boto.connect_s3(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) bucket = conn.create_bucket(bucket_name, location=boto.s3.connection.Location.DEFAULT) testfile = "replace this with an actual filename" print 'Uploading %s to Amazon S3 bucket %s' % \ (testfile, bucket_name) def percent_cb(complete, total): sys.stdout.write('.') sys.stdout.flush() k = Key(bucket) k.key = 'my test file' k.set_contents_from_filename(testfile, cb=percent_cb, num_cb=10) [UPDATE] I am not a pythonist, so thanks for the heads up about the import statements. Also, I'd not recommend placing credentials inside your own source code. If you are running this inside AWS use IAM Credentials with Instance Profiles (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html), and to keep the same behaviour in your Dev/Test environment, use something like Hologram from AdRoll (https://github.com/AdRoll/hologram)


Instruction:
How to join absolute and relative urls?

I have two urls: url1 = "http://127.0.0.1/test1/test2/test3/test5.xml" url2 = "../../test4/test6.xml" How can I get an absolute url for url2?

Output: You should use urlparse.urljoin : >>> import urlparse >>> urlparse.urljoin(url1, url2) 'http://127.0.0.1/test1/test4/test6.xml' With Python 3 (where urlparse is renamed to urllib.parse) you could use it as follows: >>> import urllib.parse >>> urllib.parse.urljoin(url1, url2) 'http://127.0.0.1/test1/test4/test6.xml'


Instruction:
Is there a way to list pip dependencies/requirements?

Without going through with the installation, I want to quickly see all the packages that pip install would install.

Output: This was tested with pip versions 8.1.2, 9.0.1, 10.0.1, and 18.1. To get the output without cluttering your current directory on Linux use pip download [package] -d /tmp --no-binary :all: -v -d tells pip the directory that download should put files in. Better, just use this script with the argument being the package name to get only the dependencies as output: #!/bin/sh PACKAGE=$1 pip download $PACKAGE -d /tmp --no-binary :all:-v 2>&1 \ | grep Collecting \ | cut -d' ' -f2 \ | grep -Ev "$PACKAGE(~|=|\!|>|<|$)" Also available here.


Instruction:
Generate random array of floats between a range

I haven't been able to find a function to generate an array of random floats of a given length between a certain range. I've looked at Random sampling but no function seems to do what I need. random.uniform comes close but it only returns a single element, not a specific number. This is what I'm after: ran_floats = some_function(low=0.5, high=13.3, size=50) which would return an array of 50 random non-unique floats (ie: repetitions are allowed) uniformly distributed in the range [0.5, 13.3]. Is there such a function?

Output: np.random.uniform fits your use case: sampl = np.random.uniform(low=0.5, high=13.3, size=(50,)) Update Oct 2019: While the syntax is still supported, it looks like the API changed with NumPy 1.17 to support greater control over the random number generator. Going forward the API has changed and you should look at https://docs.scipy.org/doc/numpy/reference/random/generated/numpy.random.Generator.uniform.html The enhancement proposal is here: https://numpy.org/neps/nep-0019-rng-policy.html


Instruction:
What is the return type hint of a generator function?

I'm trying to write a :rtype: type hint for a generator function. What is the type it returns? For example, say I have this functions which yields strings: def read_text_file(fn): """ Yields the lines of the text file one by one. :param fn: Path of text file to read. :type fn: str :rtype: ???????????????? <======================= what goes here? """ with open(fn, 'rt') as text_file: for line in text_file: yield line The return type isn't just a string, it's some kind of iterable of strings? So I can't just write :rtype: str. What's the right hint?

Output: Generator Generator[str, None, None] or Iterator[str]


Instruction:
What is the difference between class and instance attributes?

Is there any meaningful distinction between: class A(object): foo = 5 # some default value vs. class B(object): def __init__(self, foo=5): self.foo = foo If you're creating a lot of instances, is there any difference in performance or space requirements for the two styles? When you read the code, do you consider the meaning of the two styles to be significantly different?

Output: There is a significant semantic difference (beyond performance considerations): when the attribute is defined on the instance (which is what we usually do), there can be multiple objects referred to. Each gets a totally separate version of that attribute. when the attribute is defined on the class, there is only one underlying object referred to, so if operations on different instances of that class both attempt to set/(append/extend/insert/etc.) the attribute, then: if the attribute is a builtin type (like int, float, boolean, string), operations on one object will overwrite (clobber) the value if the attribute is a mutable type (like a list or a dict), we will get unwanted leakage. For example: >>> class A: foo = [] >>> a, b = A(), A() >>> a.foo.append(5) >>> b.foo [5] >>> class A: ... def __init__(self): self.foo = [] >>> a, b = A(), A() >>> a.foo.append(5) >>> b.foo []


Instruction:
What is %timeit in Python?

I always read the code to calculate the time like this way: %timeit function() What does % mean here? I think, % is always used to replace something in a string, like %s means replace a string, %d replace a data, but I have no idea about this case.

Output: %timeit is an IPython magic function, which can be used to time a particular piece of code (a single execution statement, or a single method). From the documentation: %timeit Time execution of a Python statement or expression Usage, in line mode: %timeit [-n<N> -r<R> [-t|-c] -q -p<P> -o] statement To use it, for example if we want to find out whether using xrange is any faster than using range, you can simply do: In [1]: %timeit for _ in range(1000): True 10000 loops, best of 3: 37.8 µs per loop In [2]: %timeit for _ in xrange(1000): True 10000 loops, best of 3: 29.6 µs per loop And you will get the timings for them. The major advantages of %timeit are: You don't have to import timeit.timeit from the standard library, and run the code multiple times to figure out which is the better approach. It will automatically calculate number of runs required for your code based on a total of 2 seconds execution window. You can make use of current console variables implicitly, whereas timeit.timeit requires them to be provided explicitly.


Instruction:
How to use typing.Annotated

I'm having a hard time understanding what typing.Annotated is good for from the documentation and an even harder time finding explanations/examples outside the documentation. In what context would you use Annotated? Does it depend on third-party libraries?

Output: Annotated in python allows developers to declare the type of a reference and provide additional information related to it. name: Annotated[str, "first letter is capital"] This tells that name is of type str and that name[0] is a capital letter. On its own Annotated does not do anything other than assigning extra information (metadata) to a reference. It is up to another code, which can be a library, framework or your own code, to interpret the metadata and make use of it. For example, FastAPI uses Annotated for data validation: def read_items(q: Annotated[str, Query(max_length=50)]) Here the parameter q is of type str with a maximum length of 50. This information was communicated to FastAPI (or any other underlying library) using the Annotated keyword.


Instruction:
Is there a way to compile a python application into static binary?

What I'm trying to do is ship my code to a remote server, that may have different python version installed and/or may not have packages my app requires. Right now to achieve such portability I have to build relocatable virtualenv with interpreter and code. That approach has some issues (for example, you have to manually copy a bunch of libraries into your virtualenv, since --always-copy doesn't work as expected) and generally slow. There's (in theory) a way to build python itself statically. I wonder if I could pack interpreter with my code into one binary and run my application as module. Something like that: ./mypython -m myapp run or ./mypython -m gunicorn -c ./gunicorn.conf myapp.wsgi:application.

Output: There are two ways you could go about to solve your problem Use a static builder, like freeze, or pyinstaller, or py2exe Compile using cython and link against a statically-compiled version of CPython This answer explains how you can go about doing it using the second approach, since the first method is not cross platform and version, and has been explained in other answers. Also, using programs like pyinstaller typically results in huge file sizes, while using cython will result in a file that's much smaller First, install cython. sudo -H pip3 install cython Then, you can use cython to generate a C file out of the Python .py file (in reference to https://stackoverflow.com/a/22040484/5714445) cython example_file.py --embed Use GCC to compile it after getting your statically-compiled python version (Note: The below assumes you are trying to compile it to Python3) gcc -Os $(python3-config --includes) example_file.c -o output_bin_file $(python3-config --ldflags --embed) You will now have a binary file output_bin_file, which is what you are looking for Other things to note: Change example_file.py to whatever file you are actually trying to compile. Note: Cython may not approve of filenames containing dashes (-). Cython is used to use C-Type Variable definitions for static memory allocation to speed up Python programs. In your case however, you will still be using traditional Python definitions. If you are using additional libraries (like opencv, for example), you might have to provide the directory to them using -L and then specify the name of the library using -l in the GCC Flags. For more information on this, please refer to GCC flags The above method might not work for anaconda python, as you will likely have to install a version of gcc that is compatible with your conda-python. If your Python version is not statically-compiled, this will result in a dynamic binary


Instruction:
Simple example of use of __setstate__ and __getstate__

I don't know what the __setstate__ and __getstate__ methods do, so help me with a simple example.

Output: Here's a very simple example for Python that should supplement the pickle docs. class Foo(object): def __init__(self, val=2): self.val = val def __getstate__(self): print("I'm being pickled") self.val *= 2 return self.__dict__ def __setstate__(self, d): print("I'm being unpickled with these values: " + repr(d)) self.__dict__ = d self.val *= 3 import pickle f = Foo() f_data = pickle.dumps(f) f_new = pickle.loads(f_data) Output: I'm being pickled I'm being unpickled with these values: {'val': 4}


Instruction:
random.choice from set?

I'm working on an AI portion of a guessing game. I want the AI to select a random letter from this list. I'm doing it as a set so I can easily remove letters from the list as they are guessed in the game and are therefore no longer available to be guessed again. it says set object isn't indexable. How can I work around this? import random aiTurn = True while aiTurn == True: allLetters = set(list('abcdefghijklmnopqrstuvwxyz')) aiGuess = random.choice(allLetters) print(aiGuess)

Output: Note (Oct. 2020): as of v3.9, Python has officially deprecated random.sample() working on sets, with the official guidance being to explicitly convert the set to a list or tuple before passing it in, though this doesn't solve the efficiency problems. >>> random.sample(set('abcdefghijklmnopqrstuvwxyz'), 1) ['f'] Documentation: https://docs.python.org/3/library/random.html#random.sample Note that choosing random elements from a set is extremely inefficient no matter how you do it - it takes time proportional to the size of the set, or worse if the set's underlying hash table is sparse due to removed elements. Instead, you should probably use a different data structure that supports this operation efficiently.


Instruction:
Strip / trim all strings of a dataframe

Cleaning the values of a multitype data frame in python/pandas, I want to trim the strings. I am currently doing it in two instructions : import pandas as pd df = pd.DataFrame([[' a ', 10], [' c ', 5]]) df.replace('^\s+', '', regex=True, inplace=True) #front df.replace('\s+$', '', regex=True, inplace=True) #end df.values This is quite slow, what could I improve ?

Output: You can use DataFrame.select_dtypes to select string columns and then apply function str.strip. Notice: Values cannot be types like dicts or lists, because their dtypes is object. df_obj = df.select_dtypes('object') #if need also processing string categories #df_obj = df.select_dtypes(['object', 'category']) print (df_obj) 0 a 1 c df[df_obj.columns] = df_obj.apply(lambda x: x.str.strip()) print (df) 0 1 0 a 10 1 c 5 But if there are only a few columns use str.strip: df[0] = df[0].str.strip()


Instruction:
Convert categorical data in pandas dataframe

I have a dataframe with this type of data (too many columns): col1 int64 col2 int64 col3 category col4 category col5 category Columns look like this: Name: col3, dtype: category Categories (8, object): [B, C, E, G, H, N, S, W] I want to convert all the values in each column to integer like this: [1, 2, 3, 4, 5, 6, 7, 8] I solved this for one column by this: dataframe['c'] = pandas.Categorical.from_array(dataframe.col3).codes Now I have two columns in my dataframe - old col3 and new c and need to drop old columns. That's bad practice. It works but in my dataframe there are too many columns and I don't want do it manually. How can I do this more cleverly?

Output: First, to convert a Categorical column to its numerical codes, you can do this easier with: dataframe['c'].cat.codes. Further, it is possible to select automatically all columns with a certain dtype in a dataframe using select_dtypes. This way, you can apply above operation on multiple and automatically selected columns. First making an example dataframe: In [75]: df = pd.DataFrame({'col1':[1,2,3,4,5], 'col2':list('abcab'), 'col3':list('ababb')}) In [76]: df['col2'] = df['col2'].astype('category') In [77]: df['col3'] = df['col3'].astype('category') In [78]: df.dtypes Out[78]: col1 int64 col2 category col3 category dtype: object Then by using select_dtypes to select the columns, and then applying .cat.codes on each of these columns, you can get the following result: In [80]: cat_columns = df.select_dtypes(['category']).columns In [81]: cat_columns Out[81]: Index([u'col2', u'col3'], dtype='object') In [83]: df[cat_columns] = df[cat_columns].apply(lambda x: x.cat.codes) In [84]: df Out[84]: col1 col2 col3 0 1 0 0 1 2 1 1 2 3 2 0 3 4 0 1 4 5 1 1 Note: NaN becomes -1 This method is fast because the relationship between code and category is readily available and do not need to be computed.


Instruction:
How can I split a column of tuples in a Pandas dataframe?

I have a Pandas dataframe (this is only a little piece) >>> d1 y norm test y norm train len(y_train) len(y_test) \ 0 64.904368 116.151232 1645 549 1 70.852681 112.639876 1645 549 SVR RBF \ 0 (35.652207342877873, 22.95533537448393) 1 (39.563683797747622, 27.382483096332511) LCV \ 0 (19.365430594452338, 13.880062435173587) 1 (19.099614489458364, 14.018867136617146) RIDGE CV \ 0 (4.2907610988480362, 12.416745648065584) 1 (4.18864306788194, 12.980833914392477) RF \ 0 (9.9484841581029428, 16.46902345373697) 1 (10.139848213735391, 16.282141345406522) GB \ 0 (0.012816232716538605, 15.950164822266007) 1 (0.012814519804493328, 15.305745202851712) ET DATA 0 (0.00034337162272515505, 16.284800366214057) j2m 1 (0.00024811554516431878, 15.556506191784194) j2m >>> I want to split all the columns that contain tuples. For example, I want to replace the column LCV with the columns LCV-a and LCV-b. How can I do that?

Output: You can do this by doing pd.DataFrame(col.tolist()) on that column: In [2]: df = pd.DataFrame({'a':[1,2], 'b':[(1,2), (3,4)]}) In [3]: df Out[3]: a b 0 1 (1, 2) 1 2 (3, 4) In [4]: df['b'].tolist() Out[4]: [(1, 2), (3, 4)] In [5]: pd.DataFrame(df['b'].tolist(), index=df.index) Out[5]: 0 1 0 1 2 1 3 4 In [6]: df[['b1', 'b2']] = pd.DataFrame(df['b'].tolist(), index=df.index) In [7]: df Out[7]: a b b1 b2 0 1 (1, 2) 1 2 1 2 (3, 4) 3 4 Note: in an earlier version, this answer recommended to use df['b'].apply(pd.Series) instead of pd.DataFrame(df['b'].tolist(), index=df.index). That works as well (because it makes a Series of each tuple, which is then seen as a row of a dataframe), but it is slower / uses more memory than the tolist version, as noted by the other answers here (thanks to denfromufa).


Instruction:
Python [Errno 98] Address already in use

In my Python socket program, I sometimes need to interrupt it with Ctrl-C. When I do this, it does close the connection using socket.close(). However, when I try to reopen it I have to wait what seems like a minute before I can connect again. How does one correctly close a socket? Or is this intended?

Output: Yes, this is intended. Here you can read a detailed explanation. It is possible to override this behavior by setting the SO_REUSEADDR option on a socket. For example: sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)


Instruction:
How to print out a dictionary nicely in Python?

I've just started to learn python and I'm building a text game. I want an inventory system, but I can't seem to print out the dictionary without it looking ugly. This is what I have so far: def inventory(): for numberofitems in len(inventory_content.keys()): inventory_things = list(inventory_content.keys()) inventory_amounts = list(inventory_content.values()) print(inventory_things[numberofitems])

Output: I like the pprint module (Pretty Print) included in Python. It can be used to either print the object, or format a nice string version of it. import pprint # Prints the nicely formatted dictionary pprint.pprint(dictionary) # Sets 'pretty_dict_str' to the formatted string value pretty_dict_str = pprint.pformat(dictionary) But it sounds like you are printing out an inventory, which users will likely want shown as something more like the following: def print_inventory(dct): print("Items held:") for item, amount in dct.items(): # dct.iteritems() in Python 2 print("{} ({})".format(item, amount)) inventory = { "shovels": 3, "sticks": 2, "dogs": 1, } print_inventory(inventory) which prints: Items held: shovels (3) sticks (2) dogs (1)


Instruction:
Why is the order in dictionaries and sets arbitrary?

I don't understand how looping over a dictionary or set in python is done by 'arbitrary' order. I mean, it's a programming language so everything in the language must be 100% determined, correct? Python must have some kind of algorithm that decides which part of the dictionary or set is chosen, 1st, second and so on. What am I missing?

Output: Note: This answer was written before the implementation of the dict type changed, in Python 3.6. Most of the implementation details in this answer still apply, but the listing order of keys in dictionaries is no longer determined by hash values. The set implementation remains unchanged. The order is not arbitrary, but depends on the insertion and deletion history of the dictionary or set, as well as on the specific Python implementation. For the remainder of this answer, for 'dictionary', you can also read 'set'; sets are implemented as dictionaries with just keys and no values. Keys are hashed, and hash values are assigned to slots in a dynamic table (it can grow or shrink based on needs). And that mapping process can lead to collisions, meaning that a key will have to be slotted in a next slot based on what is already there. Listing the contents loops over the slots, and so keys are listed in the order they currently reside in the table. Take the keys 'foo' and 'bar', for example, and lets assume the table size is 8 slots. In Python 2.7, hash('foo') is -4177197833195190597, hash('bar') is 327024216814240868. Modulo 8, that means these two keys are slotted in slots 3 and 4 then: >>> hash('foo') -4177197833195190597 >>> hash('foo') % 8 3 >>> hash('bar') 327024216814240868 >>> hash('bar') % 8 4 This informs their listing order: >>> {'bar': None, 'foo': None} {'foo': None, 'bar': None} All slots except 3 and 4 are empty, looping over the table first lists slot 3, then slot 4, so 'foo' is listed before 'bar'. bar and baz, however, have hash values that are exactly 8 apart and thus map to the exact same slot, 4: >>> hash('bar') 327024216814240868 >>> hash('baz') 327024216814240876 >>> hash('bar') % 8 4 >>> hash('baz') % 8 4 Their order now depends on which key was slotted first; the second key will have to be moved to a next slot: >>> {'baz': None, 'bar': None} {'bar': None, 'baz': None} >>> {'bar': None, 'baz': None} {'baz': None, 'bar': None} The table order differs here, because one or the other key was slotted first. The technical name for the underlying structure used by CPython (the most commonly used Python implemenation) is a hash table, one that uses open addressing. If you are curious, and understand C well enough, take a look at the C implementation for all the (well documented) details. You could also watch this Pycon 2010 presentation by Brandon Rhodes about how CPython dict works, or pick up a copy of Beautiful Code, which includes a chapter on the implementation written by Andrew Kuchling. Note that as of Python 3.3, a random hash seed is used as well, making hash collisions unpredictable to prevent certain types of denial of service (where an attacker renders a Python server unresponsive by causing mass hash collisions). This means that the order of a given dictionary or set is then also dependent on the random hash seed for the current Python invocation. Other implementations are free to use a different structure for dictionaries, as long as they satisfy the documented Python interface for them, but I believe that all implementations so far use a variation of the hash table. CPython 3.6 introduces a new dict implementation that maintains insertion order, and is faster and more memory efficient to boot. Rather than keep a large sparse table where each row references the stored hash value, and the key and value objects, the new implementation adds a smaller hash array that only references indices in a separate 'dense' table (one that only contains as many rows as there are actual key-value pairs), and it is the dense table that happens to list the contained items in order. See the proposal to Python-Dev for more details. Note that in Python 3.6 this is considered an implementation detail, Python-the-language does not specify that other implementations have to retain order. This changed in Python 3.7, where this detail was elevated to be a language specification; for any implementation to be properly compatible with Python 3.7 or newer it must copy this order-preserving behaviour. And to be explicit: this change doesn't apply to sets, as sets already have a 'small' hash structure. Python 2.7 and newer also provides an OrderedDict class, a subclass of dict that adds an additional data structure to record key order. At the price of some speed and extra memory, this class remembers in what order you inserted keys; listing keys, values or items will then do so in that order. It uses a doubly-linked list stored in an additional dictionary to keep the order up-to-date efficiently. See the post by Raymond Hettinger outlining the idea. OrderedDict objects have other advantages, such as being re-orderable. If you wanted an ordered set, you can install the oset package; it works on Python 2.5 and up.


Instruction:
What does Pythonic mean?

On many websites I often see comments that code isn't Pythonic, or that there is a more Pythonic way to achieve the same goal. What does Pythonic mean in this context? For example, why is while i < someValue: do_something(myList[i]) i += 1 not Pythonic while for x in myList: doSomething(x) is Pythonic?

Output: Exploiting the features of the Python language to produce code that is clear, concise and maintainable. Pythonic means code that doesn't just get the syntax right, but that follows the conventions of the Python community and uses the language in the way it is intended to be used. This is maybe easiest to explain by negative example, as in the linked article from the other answers. Examples of un-Pythonic code often come from users of other languages, who instead of learning a Python programming patterns such as list comprehensions or generator expressions, attempt to crowbar in patterns more commonly used in C or Java. Loops are particularly common examples of this. For example, in Java I might use for (int index = 0; index < items.length; index++) { items[index].performAction(); } In Python we can try and replicate this using while loops, but it would be cleaner to use: for item in items: item.perform_action() Or, even a generator expression (item.some_attribute for item in items) So essentially when someone says something is un-Pythonic, they are saying that the code could be rewritten in a way that is a better fit for Python's coding style. Typing import this at the command line gives a summary of Python principles. Less well known is that the source code for import this is decidedly, and by design, un-Pythonic! Take a look at it for an example of what not to do.


Instruction:
Efficient way to remove keys with empty strings from a dict

I have a dict and would like to remove all the keys for which there are empty value strings. metadata = {u'Composite:PreviewImage': u'(Binary data 101973 bytes)', u'EXIF:CFAPattern2': u''} What is the best way to do this?

Output: Python 2.X dict((k, v) for k, v in metadata.iteritems() if v) Python 2.7 - 3.X {k: v for k, v in metadata.items() if v} Note that all of your keys have values. It's just that some of those values are the empty string. There's no such thing as a key in a dict without a value; if it didn't have a value, it wouldn't be in the dict.


Instruction:
openpyxl - adjust column width size

I have following script which is converting a CSV file to an XLSX file, but my column size is very narrow. Each time I have to drag them with mouse to read data. Does anybody know how to set column width in openpyxl? Here is the code I am using. #!/usr/bin/python2.6 import csv from openpyxl import Workbook from openpyxl.cell import get_column_letter f = open('users_info_cvs.txt', "rU") csv.register_dialect('colons', delimiter=':') reader = csv.reader(f, dialect='colons') wb = Workbook() dest_filename = r"account_info.xlsx" ws = wb.worksheets[0] ws.title = "Users Account Information" for row_index, row in enumerate(reader): for column_index, cell in enumerate(row): column_letter = get_column_letter((column_index + 1)) ws.cell('%s%s'%(column_letter, (row_index + 1))).value = cell wb.save(filename = dest_filename)

Output: You could estimate (or use a mono width font) to achieve this. Let's assume data is a nested array like [['a1','a2'],['b1','b2']] We can get the max characters in each column. Then set the width to that. Width is exactly the width of a monospace font (if not changing other styles at least). Even if you use a variable width font it is a decent estimation. This will not work with formulas. from openpyxl.utils import get_column_letter column_widths = [] for row in data: for i, cell in enumerate(row): if len(column_widths) > i: if len(cell) > column_widths[i]: column_widths[i] = len(cell) else: column_widths += [len(cell)] for i, column_width in enumerate(column_widths,1): # ,1 to start at 1 worksheet.column_dimensions[get_column_letter(i)].width = column_width A bit of a hack but your reports will be more readable.


Instruction:
Iterate an iterator by chunks (of n) in Python?

Can you think of a nice way (maybe with itertools) to split an iterator into chunks of given size? Therefore l=[1,2,3,4,5,6,7] with chunks(l,3) becomes an iterator [1,2,3], [4,5,6], [7] I can think of a small program to do that but not a nice way with maybe itertools.

Output: The grouper() recipe from the itertools documentation's recipes comes close to what you want: def grouper(iterable, n, *, incomplete='fill', fillvalue=None): "Collect data into non-overlapping fixed-length chunks or blocks" # grouper('ABCDEFG', 3, fillvalue='x') --> ABC DEF Gxx # grouper('ABCDEFG', 3, incomplete='strict') --> ABC DEF ValueError # grouper('ABCDEFG', 3, incomplete='ignore') --> ABC DEF args = [iter(iterable)] * n if incomplete == 'fill': return zip_longest(*args, fillvalue=fillvalue) if incomplete == 'strict': return zip(*args, strict=True) if incomplete == 'ignore': return zip(*args) else: raise ValueError('Expected fill, strict, or ignore') This won't work well when the last chunk is incomplete though, as, depending on the incomplete mode, it will either fill up the last chunk with a fill value, raise an exception, or silently drop the incomplete chunk. In more recent versions of the recipes they added the batched recipe that does exactly what you want: def batched(iterable, n): "Batch data into tuples of length n. The last batch may be shorter." # batched('ABCDEFG', 3) --> ABC DEF G if n < 1: raise ValueError('n must be at least one') it = iter(iterable) while (batch := tuple(islice(it, n))): yield batch Finally, a less general solution that only works on sequences but does handle the last chunk as desired and preserves the type of the original sequence is: (my_list[i:i + chunk_size] for i in range(0, len(my_list), chunk_size)) Since python 3.12, you can also just use itertools.batched. From docs: itertools.batched(iterable, n) Batch data from the iterable into tuples of length n. The last batch may be shorter than n.


Instruction:
Creating a zero-filled pandas data frame

What is the best way to create a zero-filled pandas data frame of a given size? I have used: zero_data = np.zeros(shape=(len(data),len(feature_list))) d = pd.DataFrame(zero_data, columns=feature_list) Is there a better way to do it?

Output: Create and fill a pandas dataframe with zeros feature_list = ["foo", "bar", 37] df = pd.DataFrame(0, index=np.arange(7), columns=feature_list) print(df) which prints: foo bar 37 0 0 0 0 1 0 0 0 2 0 0 0 3 0 0 0 4 0 0 0 5 0 0 0 6 0 0 0


Instruction:
How can I install from a git subdirectory with pip?

I have a git repository with many folders, one of them being a python module installable with pip, like this: repo.git/ repo.git/folder1/ repo.git/folder2/ repo.git/mymodule/ repo.git/mymodule/__init__.py repo.git/mymodule/setup.py repo.git/mymodule/... Right now I have to do the following to install: git clone http://server/repo.git cd repo pip install mymodule cd .. rm -rf repo Is it possible to install the module directly with pip without explicitly cloning ? I tried: pip install git+https://server/repo.git/mymodule/ pip install git+https://server/repo.git:mymodule/ But I get: IOError: [Errno 2] No such file or directory: '/tmp/pip-88tlLm-build/setup.py'

Output: There is a pull request regarding this feature, and it seems to have been merged to develop branch a month ago. The syntax is the following: pip install -e "git+https://git.repo/some_repo.git#egg=$NAME_OF_PACKAGE&subdirectory=$SUBDIR_IN_REPO" # install a python package from a repo subdirectory We probably have to wait for a while until it gets merged to master and is distributed. UPDATE: This is now available and documented at https://pip.pypa.io/en/stable/cli/pip_install/#vcs-support as follows: For projects where setup.py is not in the root of project, "subdirectory" component is used. Value of "subdirectory" component should be a path starting from root of the project to where setup.py is located. So if your repository layout is: - pkg_dir/ - setup.py # setup.py for package ``pkg`` - some_module.py - other_dir/ - some_file - some_other_file You'll need to use pip install -e "vcs+protocol://repo_url/#egg=pkg&subdirectory=pkg_dir" Note: Make sure to surround it with quotes otherwise subdirectory won't work.


Instruction:
Regular expression matching a multiline block of text

I'm having a bit of trouble getting a Python regex to work when matching against text that spans multiple lines. The example text is (\n is a newline) some Varying TEXT\n \n DSJFKDAFJKDAFJDSAKFJADSFLKDLAFKDSAF\n [more of the above, ending with a newline]\n [yep, there is a variable number of lines here]\n \n (repeat the above a few hundred times). I'd like to capture two things: the some Varying TEXT part all lines of uppercase text that come two lines below it in one capture (I can strip out the newline characters later). I've tried a few approaches: re.compile(r"^>(\w+)$$([.$]+)^$", re.MULTILINE) # try to capture both parts re.compile(r"(^[^>][\w\s]+)$", re.MULTILINE|re.DOTALL) # just textlines ...and a lot of variations hereof with no luck. The last one seems to match the lines of text one by one, which is not what I really want. I can catch the first part, no problem, but I can't seem to catch the 4-5 lines of uppercase text. I'd like match.group(1) to be some Varying Text and group(2) to be line1+line2+line3+etc until the empty line is encountered. If anyone's curious, it's supposed to be a sequence of amino acids that make up a protein.

Output: Try this: re.compile(r"^(.+)\n((?:\n.+)+)", re.MULTILINE) I think your biggest problem is that you're expecting the ^ and $ anchors to match linefeeds, but they don't. In multiline mode, ^ matches the position immediately following a newline and $ matches the position immediately preceding a newline. Be aware, too, that a newline can consist of a linefeed (\n), a carriage-return (\r), or a carriage-return+linefeed (\r\n). If you aren't certain that your target text uses only linefeeds, you should use this more inclusive version of the regex: re.compile(r"^(.+)(?:\n|\r\n?)((?:(?:\n|\r\n?).+)+)", re.MULTILINE) BTW, you don't want to use the DOTALL modifier here; you're relying on the fact that the dot matches everything except newlines.


Instruction:
How to toggle a value?

What is the most efficient way to toggle between 0 and 1?

Output: Solution using NOT If the values are boolean, the fastest approach is to use the not operator: >>> x = True >>> x = not x # toggle >>> x False >>> x = not x # toggle >>> x True >>> x = not x # toggle >>> x False Solution using subtraction If the values are numerical, then subtraction from the total is a simple and fast way to toggle values: >>> A = 5 >>> B = 3 >>> total = A + B >>> x = A >>> x = total - x # toggle >>> x 3 >>> x = total - x # toggle >>> x 5 >>> x = total - x # toggle >>> x 3 Solution using XOR If the value toggles between 0 and 1, you can use a bitwise exclusive-or: >>> x = 1 >>> x ^= 1 >>> x 0 >>> x ^= 1 >>> x 1 The technique generalizes to any pair of integers. The xor-by-one step is replaced with a xor-by-precomputed-constant: >>> A = 205 >>> B = -117 >>> t = A ^ B # precomputed toggle constant >>> x = A >>> x ^= t # toggle >>> x -117 >>> x ^= t # toggle >>> x 205 >>> x ^= t # toggle >>> x -117 (This idea was submitted by Nick Coghlan and later generalized by @zxxc.) Solution using a dictionary If the values are hashable, you can use a dictionary: >>> A = 'xyz' >>> B = 'pdq' >>> d = {A:B, B:A} >>> x = A >>> x = d[x] # toggle >>> x 'pdq' >>> x = d[x] # toggle >>> x 'xyz' >>> x = d[x] # toggle >>> x 'pdq' Solution using a conditional expression The slowest way is to use a conditional expression: >>> A = [1,2,3] >>> B = [4,5,6] >>> x = A >>> x = B if x == A else A >>> x [4, 5, 6] >>> x = B if x == A else A >>> x [1, 2, 3] >>> x = B if x == A else A >>> x [4, 5, 6] Solution using itertools If you have more than two values, the itertools.cycle() function provides a generic fast way to toggle between successive values: >>> import itertools >>> toggle = itertools.cycle(['red', 'green', 'blue']).__next__ >>> toggle() 'red' >>> toggle() 'green' >>> toggle() 'blue' >>> toggle() 'red' >>> toggle() 'green' >>> toggle() 'blue'


Instruction:
String-based enum in Python

To encapsulate a list of states I am using enum module: from enum import Enum class MyEnum(Enum): state1='state1' state2 = 'state2' state = MyEnum.state1 MyEnum['state1'] == state # here it works 'state1' == state # here it does not throw but returns False (fail!) However, the issue is that I need to seamlessly use the values as strings in many contexts in my script, like: select_query1 = select(...).where(Process.status == str(MyEnum.state1)) # works but ugly select_query2 = select(...).where(Process.status == MyEnum.state1) # throws exeption How to do it avoiding calling additional type conversion (str(state) above) or the underlying value (state.value)?

Output: It seems that it is enough to inherit from str class at the same time as Enum: from enum import Enum class MyEnum(str, Enum): state1 = 'state1' state2 = 'state2' The tricky part is that the order of classes in the inheritance chain is important as this: class MyEnum(Enum, str): state1 = 'state1' state2 = 'state2' throws: TypeError: new enumerations should be created as `EnumName([mixin_type, ...] [data_type,] enum_type)` With the correct class the following operations on MyEnum are fine: print('This is the state value: ' + state) As a side note, it seems that the special inheritance trick is not needed for formatted strings which work even for Enum inheritance only: msg = f'This is the state value: {state}' # works without inheriting from str


Instruction:
How to schedule a function to run every hour on Flask?

I have a Flask web hosting with no access to cron command. How can I execute some Python function every hour?

Output: You can use BackgroundScheduler() from APScheduler package (v3.5.3): import time import atexit from apscheduler.schedulers.background import BackgroundScheduler def print_date_time(): print(time.strftime("%A, %d. %B %Y %I:%M:%S %p")) scheduler = BackgroundScheduler() scheduler.add_job(func=print_date_time, trigger="interval", seconds=60) scheduler.start() # Shut down the scheduler when exiting the app atexit.register(lambda: scheduler.shutdown()) Note that two of these schedulers will be launched when Flask is in debug mode. For more information, check out this question.


Instruction:
Rank items in an array using Python/NumPy, without sorting array twice

I have an array of numbers and I'd like to create another array that represents the rank of each item in the first array. I'm using Python and NumPy. For example: array = [4,2,7,1] ranks = [2,1,3,0] Here's the best method I've come up with: array = numpy.array([4,2,7,1]) temp = array.argsort() ranks = numpy.arange(len(array))[temp.argsort()] Are there any better/faster methods that avoid sorting the array twice?

Output: Use advanced indexing on the left-hand side in the last step: array = numpy.array([4,2,7,1]) temp = array.argsort() ranks = numpy.empty_like(temp) ranks[temp] = numpy.arange(len(array)) This avoids sorting twice by inverting the permutation in the last step.


Instruction:
OperationalError: database is locked

I have made some repetitive operations in my application (testing it), and suddenly I’m getting a weird error: OperationalError: database is locked I've restarted the server, but the error persists. What can it be all about?

Output: From Django's documentation: SQLite is meant to be a lightweight database, and thus can't support a high level of concurrency. OperationalError: database is locked errors indicate that your application is experiencing more concurrency than sqlite can handle in default configuration. This error means that one thread or process has an exclusive lock on the database connection and another thread timed out waiting for the lock the be released. Python's SQLite wrapper has a default timeout value that determines how long the second thread is allowed to wait on the lock before it times out and raises the OperationalError: database is locked error. If you're getting this error, you can solve it by: Switching to another database backend. At a certain point SQLite becomes too "lite" for real-world applications, and these sorts of concurrency errors indicate you've reached that point. Rewriting your code to reduce concurrency and ensure that database transactions are short-lived. Increase the default timeout value by setting the timeout database option: "OPTIONS": { # ... "timeout": 20, # ... } This will make SQLite wait a bit longer before throwing "database is locked" errors; it won't really do anything to solve them.


Instruction:
Is it possible to have multiple statements in a python lambda expression?

I have a list of lists: lst = [[567, 345, 234], [253, 465, 756, 2345], [333, 777, 111, 555]] I want map lst into another list containing only the second smallest number from each sublist. So the result should be [345, 465, 333]. If I were just interested in the smallest number, I could write this as map(lambda x: min(x), lst). To get the second smallest numbers, I thought of sorting and then indexing the results, like map(lambda x: sort(x)[1], lst); however, sort returns None rather than chaining. If it were possible to use multiple statements in a lambda, I could write map(lambda x: sort(x); x[1], lst), but this is not allowed. Can I use map to solve the problem without defining a named function? How?

Output: There are several different answers I can give here, from your specific question to more general concerns. So from most specific to most general: Q. Can you put multiple statements in a lambda? A. No. But you don't actually need to use a lambda. You can put the statements in a def instead. i.e.: def second_lowest(l): l.sort() return l[1] map(second_lowest, lst) Q. Can you get the second lowest item from a lambda by sorting the list? A. Yes. As alex's answer points out, sorted() is a version of sort that creates a new list, rather than sorting in-place, and can be chained. Note that this is probably what you should be using - it's bad practice for your map to have side effects on the original list. Q. How should I get the second lowest item from each list in a sequence of lists? A. sorted(l)[1] is not actually the best way for this. It has O(N log(N)) complexity, while an O(n) solution exists. This can be found in the heapq module. >>> import heapq >>> l = [5,2,6,8,3,5] >>> heapq.nsmallest(l, 2) [2, 3] So just use: map(lambda x: heapq.nsmallest(x,2)[1], list_of_lists) It's also usually considered clearer to use a list comprehension, which avoids the lambda altogether: [heapq.nsmallest(x,2)[1] for x in list_of_lists]


Instruction:
In Python script, how do I set PYTHONPATH?

I know how to set it in my /etc/profile and in my environment variables. But what if I want to set it during a script? Is it import os, sys? How do I do it?

Output: You don't set PYTHONPATH, you add entries to sys.path. It's a list of directories that should be searched for Python packages, so you can just append your directories to that list. import sys sys.path.append('/path/to/whatever') In fact, sys.path is initialized by splitting the value of PYTHONPATH on the path separator character (: on Linux-like systems, ; on Windows). You can also add directories using site.addsitedir, and that method will also take into account .pth files existing within the directories you pass. (That would not be the case with directories you specify in PYTHONPATH.)


Instruction:
Convert a bytes array into JSON format

I want to parse a bytes string in JSON format to convert it into python objects. This is the source I have: my_bytes_value = b'[{\'Date\': \'2016-05-21T21:35:40Z\', \'CreationDate\': \'2012-05-05\', \'LogoType\': \'png\', \'Ref\': 164611595, \'Classe\': [\'Email addresses\', \'Passwords\'],\'Link\':\'http://some_link.com\'}]' And this is the desired outcome I want to have: [{ "Date": "2016-05-21T21:35:40Z", "CreationDate": "2012-05-05", "LogoType": "png", "Ref": 164611595, "Classes": [ "Email addresses", "Passwords" ], "Link": "http://some_link.com"}] First, I converted the bytes to string: my_new_string_value = my_bytes_value.decode("utf-8") but when I try to invoke loads to parse it as JSON: my_json = json.loads(my_new_string_value) I get this error: json.decoder.JSONDecodeError: Expecting value: line 1 column 174 (char 173)

Output: Your bytes object is almost JSON, but it's using single quotes instead of double quotes, and it needs to be a string. So one way to fix it is to decode the bytes to str and replace the quotes. Another option is to use ast.literal_eval; see below for details. If you want to print the result or save it to a file as valid JSON you can load the JSON to a Python list and then dump it out. Eg, import json my_bytes_value = b'[{\'Date\': \'2016-05-21T21:35:40Z\', \'CreationDate\': \'2012-05-05\', \'LogoType\': \'png\', \'Ref\': 164611595, \'Classe\': [\'Email addresses\', \'Passwords\'],\'Link\':\'http://some_link.com\'}]' # Decode UTF-8 bytes to Unicode, and convert single quotes # to double quotes to make it valid JSON my_json = my_bytes_value.decode('utf8').replace("'", '"') print(my_json) print('- ' * 20) # Load the JSON to a Python list & dump it back out as formatted JSON data = json.loads(my_json) s = json.dumps(data, indent=4, sort_keys=True) print(s) output [{"Date": "2016-05-21T21:35:40Z", "CreationDate": "2012-05-05", "LogoType": "png", "Ref": 164611595, "Classe": ["Email addresses", "Passwords"],"Link":"http://some_link.com"}] - - - - - - - - - - - - - - - - - - - - [ { "Classe": [ "Email addresses", "Passwords" ], "CreationDate": "2012-05-05", "Date": "2016-05-21T21:35:40Z", "Link": "http://some_link.com", "LogoType": "png", "Ref": 164611595 } ] As Antti Haapala mentions in the comments, we can use ast.literal_eval to convert my_bytes_value to a Python list, once we've decoded it to a string. from ast import literal_eval import json my_bytes_value = b'[{\'Date\': \'2016-05-21T21:35:40Z\', \'CreationDate\': \'2012-05-05\', \'LogoType\': \'png\', \'Ref\': 164611595, \'Classe\': [\'Email addresses\', \'Passwords\'],\'Link\':\'http://some_link.com\'}]' data = literal_eval(my_bytes_value.decode('utf8')) print(data) print('- ' * 20) s = json.dumps(data, indent=4, sort_keys=True) print(s) Generally, this problem arises because someone has saved data by printing its Python repr instead of using the json module to create proper JSON data. If it's possible, it's better to fix that problem so that proper JSON data is created in the first place.


Instruction:
Joining pandas DataFrames by Column names

I have two DataFrames with the following column names: frame_1: event_id, date, time, county_ID frame_2: countyid, state I would like to get a DataFrame with the following columns by left-joining on county_ID = countyid: joined_dataframe: event_id, date, time, county, state I cannot figure out how to do it if the columns on which I want to join are not the index.

Output: You can use the left_on and right_on options of pd.merge as follows: pd.merge(frame_1, frame_2, left_on='county_ID', right_on='countyid') Or equivalently with DataFrame.merge: frame_1.merge(frame_2, left_on='county_ID', right_on='countyid') I was not sure from the question if you only wanted to merge if the key was in the left hand DataFrame. If that is the case then the following will do that (the above will in effect do a many to many merge) pd.merge(frame_1, frame_2, how='left', left_on='county_ID', right_on='countyid') Or frame_1.merge(frame_2, how='left', left_on='county_ID', right_on='countyid')


Instruction:
How to convert single-row pandas data frame to series?

I'm somewhat new to pandas. I have a pandas data frame that is 1 row by 23 columns. I want to convert this into a series. I'm wondering what the most pythonic way to do this is? I've tried pd.Series(myResults) but it complains ValueError: cannot copy sequence with size 23 to array axis with dimension 1. It's not smart enough to realize it's still a "vector" in math terms.

Output: If you have a one column dataframe df, you can convert it to a series: df.iloc[:,0] # pandas Series Since you have a one row dataframe df, you can transpose it so you're in the previous case: df.T.iloc[:,0]


Instruction:
Why import seaborn as sns?

Why do you always import seaborn as sns and not with the letters of the name as sbn? Is sns an acronym for something? Or is it some kind of joke? As cel commented, someone put this as an issue in github. There, they do not answer the question but say that it is a joke related to the original name of the library.

Output: Samuel Norman "Sam" Seaborn is a fictional character portrayed by Rob Lowe on the television serial drama The West Wing. So, it's a joked initialism. (source: seaborn FAQ)


Instruction:
Why am I getting ImportError: No module named pip ' right after installing pip?

I have installed pip and ez setup. I also checked the system path and I can see the module in the folder structure. Still when I try to run the pip command, I get an Import error saying no module named pip. I am running 32-bit Python on a Windows 7 machine.

Output: Just be sure that you have included the Python executable in the Windows Environment Variables, System PATH variable, and then run: python -m ensurepip. or if you have permission issue, run: python -m ensurepip --user


Instruction:
Quick and easy file dialog in Python?

I have a simple script which parses a file and loads it's contents to a database. I don't need a UI, but right now I'm prompting the user for the file to parse using raw_input which is most unfriendly, especially because the user can't copy/paste the path. I would like a quick and easy way to present a file selection dialog to the user, they can select the file, and then it's loaded to the database. (In my use case, if they happened to chose the wrong file, it would fail parsing, and wouldn't be a problem even if it was loaded to the database.) import tkFileDialog file_path_string = tkFileDialog.askopenfilename() This code is close to what I want, but it leaves an annoying empty frame open (which isn't able to be closed, probably because I haven't registered a close event handler). I don't have to use tkInter, but since it's in the Python standard library it's a good candidate for quickest and easiest solution. Whats a quick and easy way to prompt for a file or filename in a script without any other UI?

Output: Tkinter is the easiest way if you don't want to have any other dependencies. To show only the dialog without any other GUI elements, you have to hide the root window using the withdraw method: import tkinter as tk from tkinter import filedialog root = tk.Tk() root.withdraw() file_path = filedialog.askopenfilename() Python 2 variant: import Tkinter, tkFileDialog root = Tkinter.Tk() root.withdraw() file_path = tkFileDialog.askopenfilename()


Instruction:
ImproperlyConfiguredError about app_name when using namespace in include()

I am currently trying out Django. I use the namespace argument in one of my include()s in urls.py. When I run the server and try to browse, I get this error. File "C:\Users\User\AppData\Local\Programs\Python\Python36-32\lib\site-packages\django\urls\conf.py", line 39, in include 'Specifying a namespace in include() without providing an app_name ' django.core.exceptions.ImproperlyConfigured: Specifying a namespace in include() without providing an app_name is not supported. Set the app_name attribute in the included module, or pass a 2-tuple containing the list of patterns and app_name instead. These are my urls.py files: #project/urls.py from django.conf.urls import include, url from django.contrib import admin urlpatterns = [ url(r'^reviews/', include('reviews.urls', namespace='reviews')), url(r'^admin/', include(admin.site.urls)), ] and #app/urls.py from django.conf.urls import url from . import views urlpatterns = [ # ex: / url(r'^$', views.review_list, name='review_list'), # ex: /review/5/ url(r'^review/(?P<review_id>[0-9]+)/$', views.review_detail, name='review_detail'), # ex: /wine/ url(r'^wine$', views.wine_list, name='wine_list'), # ex: /wine/5/ url(r'^wine/(?P<wine_id>[0-9]+)/$', views.wine_detail, name='wine_detail'), ] What do I pass the app_name as stated in the error message?

Output: Check the docs for include here. What you've done is not an acceptable way of passing parameters to include. You could do: url(r'^reviews/', include(('reviews.urls', 'reviews'), namespace='reviews')),