Instruction:
Why is a list comprehension so much faster than appending to a list?

I was wondering why list comprehension is so much faster than appending to a list. I thought the difference is just expressive, but it's not. >>> import timeit >>> timeit.timeit(stmt='''\ t = [] for i in range(10000): t.append(i)''', number=10000) 9.467898777974142 >>> timeit.timeit(stmt='t= [i for i in range(10000)]', number=10000) 4.1138417314859 The list comprehension is 50% faster. Why?

Output: List comprehension is basically just a "syntactic sugar" for the regular for loop. In this case the reason that it performs better is because it doesn't need to load the append attribute of the list and call it as a function at each iteration. In other words and in general, list comprehensions perform faster because suspending and resuming a function's frame, or multiple functions in other cases, is slower than creating a list on demand. Consider the following examples : In [1]: def f1(): ...: l = [] ...: for i in range(5): ...: l.append(i) ...: ...: ...: def f2(): ...: [i for i in range(5)] ...: In [3]: import dis In [4]: dis.dis(f1) 2 0 BUILD_LIST 0 2 STORE_FAST 0 (l) 3 4 LOAD_GLOBAL 0 (range) 6 LOAD_CONST 1 (5) 8 CALL_FUNCTION 1 10 GET_ITER >> 12 FOR_ITER 14 (to 28) 14 STORE_FAST 1 (i) 4 16 LOAD_FAST 0 (l) 18 LOAD_METHOD 1 (append) 20 LOAD_FAST 1 (i) 22 CALL_METHOD 1 24 POP_TOP 26 JUMP_ABSOLUTE 12 >> 28 LOAD_CONST 0 (None) 30 RETURN_VALUE In [5]: In [5]: dis.dis(f2) 8 0 LOAD_CONST 1 (<code object <listcomp> at 0x7f397abc0d40, file "<ipython-input-1-45c11e415ee9>", line 8>) 2 LOAD_CONST 2 ('f2.<locals>.<listcomp>') 4 MAKE_FUNCTION 0 6 LOAD_GLOBAL 0 (range) 8 LOAD_CONST 3 (5) 10 CALL_FUNCTION 1 12 GET_ITER 14 CALL_FUNCTION 1 16 POP_TOP 18 LOAD_CONST 0 (None) 20 RETURN_VALUE Disassembly of <code object <listcomp> at 0x7f397abc0d40, file "<ipython-input-1-45c11e415ee9>", line 8>: 8 0 BUILD_LIST 0 2 LOAD_FAST 0 (.0) >> 4 FOR_ITER 8 (to 14) 6 STORE_FAST 1 (i) 8 LOAD_FAST 1 (i) 10 LIST_APPEND 2 12 JUMP_ABSOLUTE 4 >> 14 RETURN_VALUE In [6]: You can see that on offset 18 in the first function we have an append attribute while there's no such thing in second function using list comprehension. All those extra bytecodes will make the appending approach slower and since in this case you'll have loading of the append attribute in each iteration, in the end it will make the code to take approximately twice as slower as the second function using only list comprehension.


Instruction:
How do I specify multiple types for a parameter using type-hints?

I have a Python function which accepts XML data as an str. For convenience, the function also checks for xml.etree.ElementTree.Element and will automatically convert to str if necessary. import xml.etree.ElementTree as ET def post_xml(data: str): if type(data) is ET.Element: data = ET.tostring(data).decode() # ... Is it possible to specify with type-hints that a parameter can be given as one of two types? def post_xml(data: str or ET.Element): # ...

Output: You want a type union. For Python 3.10+: def post_xml(data: str | ET.Element): For earlier versions: from typing import Union def post_xml(data: Union[str, ET.Element]): ...


Instruction:
Is there a way to make a discontinuous axis in Matplotlib?

I'm trying to create a plot using pyplot that has a discontinuous x-axis. The usual way this is drawn is that the axis will have something like this: (values)----//----(later values) where the // indicates that you're skipping everything between (values) and (later values). I haven't been able to find any examples of this, so I'm wondering if it's even possible. I know you can join data over a discontinuity for, eg, financial data, but I'd like to make the jump in the axis more explicit. At the moment I'm just using subplots but I'd really like to have everything end up on the same graph in the end.

Output: Paul's answer is a perfectly fine method of doing this. However, if you don't want to make a custom transform, you can just use two subplots to create the same effect. Rather than put together an example from scratch, there's an excellent example of this written by Paul Ivanov in the matplotlib examples (It's only in the current git tip, as it was only committed a few months ago. It's not on the webpage yet.). This is just a simple modification of this example to have a discontinuous x-axis instead of the y-axis. (Which is why I'm making this post a CW) Basically, you just do something like this: import matplotlib.pylab as plt import numpy as np # If you're not familiar with np.r_, don't worry too much about this. It's just # a series with points from 0 to 1 spaced at 0.1, and 9 to 10 with the same spacing. x = np.r_[0:1:0.1, 9:10:0.1] y = np.sin(x) fig,(ax,ax2) = plt.subplots(1, 2, sharey=True) # plot the same data on both axes ax.plot(x, y, 'bo') ax2.plot(x, y, 'bo') # zoom-in / limit the view to different portions of the data ax.set_xlim(0,1) # most of the data ax2.set_xlim(9,10) # outliers only # hide the spines between ax and ax2 ax.spines['right'].set_visible(False) ax2.spines['left'].set_visible(False) ax.yaxis.tick_left() ax.tick_params(labeltop='off') # don't put tick labels at the top ax2.yaxis.tick_right() # Make the spacing between the two axes a bit smaller plt.subplots_adjust(wspace=0.15) plt.show() To add the broken axis lines // effect, we can do this (again, modified from Paul Ivanov's example): import matplotlib.pylab as plt import numpy as np # If you're not familiar with np.r_, don't worry too much about this. It's just # a series with points from 0 to 1 spaced at 0.1, and 9 to 10 with the same spacing. x = np.r_[0:1:0.1, 9:10:0.1] y = np.sin(x) fig,(ax,ax2) = plt.subplots(1, 2, sharey=True) # plot the same data on both axes ax.plot(x, y, 'bo') ax2.plot(x, y, 'bo') # zoom-in / limit the view to different portions of the data ax.set_xlim(0,1) # most of the data ax2.set_xlim(9,10) # outliers only # hide the spines between ax and ax2 ax.spines['right'].set_visible(False) ax2.spines['left'].set_visible(False) ax.yaxis.tick_left() ax.tick_params(labeltop='off') # don't put tick labels at the top ax2.yaxis.tick_right() # Make the spacing between the two axes a bit smaller plt.subplots_adjust(wspace=0.15) # This looks pretty good, and was fairly painless, but you can get that # cut-out diagonal lines look with just a bit more work. The important # thing to know here is that in axes coordinates, which are always # between 0-1, spine endpoints are at these locations (0,0), (0,1), # (1,0), and (1,1). Thus, we just need to put the diagonals in the # appropriate corners of each of our axes, and so long as we use the # right transform and disable clipping. d = .015 # how big to make the diagonal lines in axes coordinates # arguments to pass plot, just so we don't keep repeating them kwargs = dict(transform=ax.transAxes, color='k', clip_on=False) ax.plot((1-d,1+d),(-d,+d), **kwargs) # top-left diagonal ax.plot((1-d,1+d),(1-d,1+d), **kwargs) # bottom-left diagonal kwargs.update(transform=ax2.transAxes) # switch to the bottom axes ax2.plot((-d,d),(-d,+d), **kwargs) # top-right diagonal ax2.plot((-d,d),(1-d,1+d), **kwargs) # bottom-right diagonal # What's cool about this is that now if we vary the distance between # ax and ax2 via f.subplots_adjust(hspace=...) or plt.subplot_tool(), # the diagonal lines will move accordingly, and stay right at the tips # of the spines they are 'breaking' plt.show()


Instruction:
Virtual environment in R?

I've found several posts about best practice, reproducibility and workflow in R, for example: How to increase longer term reproducibility of research (particularly using R and Sweave) Complete substantive examples of reproducible research using R One of the major preoccupations is ensuring portability of code, in the sense that moving it to a new machine (possibly running a different OS) is relatively straightforward and gives the same results. Coming from a Python background, I'm used to the concept of a virtual environment. When coupled with a simple list of required packages, this goes some way to ensuring that the installed packages and libraries are available on any machine without too much fuss. Sure, it's no guarantee - different OSes have their own foibles and peculiarities - but it gets you 95% of the way there. Does such a thing exist within R? Even if it's not as sophisticated. For example simply maintaining a plain text list of required packages and a script that will install any that are missing? I'm about to start using R in earnest for the first time, probably in conjunction with Sweave, and would ideally like to start in the best way possible! Thanks for your thoughts.

Output: It looks like there is another option from RStudio devs, renv. It's available on CRAN and supersedes Packrat. In short, you use renv::init() to initialize your project library, and use renv::snapshot() / renv::restore() to save and load the state of your library. I prefer this option to conda r-enviroments because here everything is stored in the file renv.lock, which can be committed to a Git repo and distributed to the team.


Instruction:
How to open file using argparse?

I want to open file for reading using argparse. In cmd it must look like: my_program.py /filepath That's my try: parser = argparse.ArgumentParser() parser.add_argument('file', type = file) args = parser.parse_args() This gives me [edit: added later from comment by OP]: parser.add_argument('file', type = file) NameError: name 'file' is not defined

Output: The type of the argument should be string (which is default anyway). So make it like this: parser = argparse.ArgumentParser() parser.add_argument('filename') args = parser.parse_args() with open(args.filename) as file: # do stuff here


Instruction:
Use virtualenv with Python with Visual Studio Code

I have a Python project and navigation/autocomplete work perfectly within files in the project. I want to specify a virtualenv so that navigation/autocomplete work with packages inside that virtualenv. I tried this in settings.py, but navigation/autocomplete does not work. Also by setting "python.pythonPath":"~/dev/venvs/proj/bin/python killed the original navigation/autocomplete. { "python.autoComplete.extraPaths": [ "~/dev/venvs/proj", "~/dev/venvs/proj/lib" ] }

Output: With the latest update to the extension all you need to do is just specify the "python.pythonPath" as follows. The values for "python.autoComplete.extraPaths" will be determined during runtime, but you are still free to specify custom paths in there. Please, remember to restart Visual Studio Code once the necessary changes have been made. { "python.pythonPath": "~/dev/venvs/proj/bin/python" }


Instruction:
How to declare variable type, C style in Python

I'm a programming student and my teacher is starting with C to teach us the programming paradigms, he said it's ok if I deliver my homework in python (it's easier and faster for the homeworks). And I would like to have my code to be as close as possible as in plain C. Question is: How do I declare data types for variables in python like you do in C. ex: int X,Y,Z; I know I can do this in python: x = 0 y = 0 z = 0 But that seems a lot of work and it misses the point of python being easier/faster than C. So, whats the shortest way to do this? P.S. I know you don't have to declare the data type in python most of the time, but still I would like to do it so my code looks as much possible like classmates'.

Output: Edit: Python 3.5 introduced type hints which introduced a way to specify the type of a variable. This answer was written before this feature became available. There is no way to declare variables in Python, since neither "declaration" nor "variables" in the C sense exist. This will bind the three names to the same object: x = y = z = 0


Instruction:
How to aggregate unique count with pandas pivot_table

This code: df2 = ( pd.DataFrame({ 'X' : ['X1', 'X1', 'X1', 'X1'], 'Y' : ['Y2', 'Y1', 'Y1', 'Y1'], 'Z' : ['Z3', 'Z1', 'Z1', 'Z2'] }) ) g = df2.groupby('X') pd.pivot_table(g, values='X', rows='Y', cols='Z', margins=False, aggfunc='count') returns the following error: Traceback (most recent call last): ... AttributeError: 'Index' object has no attribute 'index' How do I get a Pivot Table with counts of unique values of one DataFrame column for two other columns? Is there aggfunc for count unique? Should I be using np.bincount()? NB. I am aware of pandas.Series.values_counts() however I need a pivot table. EDIT: The output should be: Z Z1 Z2 Z3 Y Y1 1 1 NaN Y2 NaN NaN 1

Output: Do you mean something like this? >>> df2.pivot_table(values='X', index='Y', columns='Z', aggfunc=lambda x: len(x.unique())) Z Z1 Z2 Z3 Y Y1 1 1 NaN Y2 NaN NaN 1 Note that using len assumes you don't have NAs in your DataFrame. You can do x.value_counts().count() or len(x.dropna().unique()) otherwise.


Instruction:
Python reading from a file and saving to utf-8

I'm having problems reading from a file, processing its string and saving to an UTF-8 File. Here is the code: try: filehandle = open(filename,"r") except: print("Could not open file " + filename) quit() text = filehandle.read() filehandle.close() I then do some processing on the variable text. And then try: writer = open(output,"w") except: print("Could not open file " + output) quit() #data = text.decode("iso 8859-15") #writer.write(data.encode("UTF-8")) writer.write(text) writer.close() This output the file perfectly but it does so in iso 8859-15 according to my editor. Since the same editor recognizes the input file (in the variable filename) as UTF-8 I don't know why this happened. As far as my reasearch has shown the commented lines should solve the problem. However when I use those lines the resulting file has gibberish in special character mainly, words with tilde as the text is in spanish. I would really appreciate any help as I am stumped....

Output: Process text to and from Unicode at the I/O boundaries of your program using open with the encoding parameter. Make sure to use the (hopefully documented) encoding of the file being read. The default encoding varies by OS (specifically, locale.getpreferredencoding(False) is the encoding used), so I recommend always explicitly using the encoding parameter for portability and clarity (Python 3 syntax below): with open(filename, 'r', encoding='utf8') as f: text = f.read() # process Unicode text with open(filename, 'w', encoding='utf8') as f: f.write(text) If still using Python 2 or for Python 2/3 compatibility, the io module implements open with the same semantics as Python 3's open and exists in both versions: import io with io.open(filename, 'r', encoding='utf8') as f: text = f.read() # process Unicode text with io.open(filename, 'w', encoding='utf8') as f: f.write(text)


Instruction:
AttributeError: Can only use .dt accessor with datetimelike values

Hi I am using pandas to convert a column to month. When I read my data they are objects: Date object dtype: object So I am first making them to date time and then try to make them as months: import pandas as pd file = '/pathtocsv.csv' df = pd.read_csv(file, sep = ',', encoding='utf-8-sig', usecols= ['Date', 'ids']) df['Date'] = pd.to_datetime(df['Date']) df['Month'] = df['Date'].dt.month Also if that helps: In [10]: df['Date'].dtype Out[10]: dtype('O') So, the error I get is like this: /Library/Frameworks/Python.framework/Versions/2.7/bin/User/lib/python2.7/site-packages/pandas/core/series.pyc in _make_dt_accessor(self) 2526 return maybe_to_datetimelike(self) 2527 except Exception: -> 2528 raise AttributeError("Can only use .dt accessor with datetimelike " 2529 "values") 2530 AttributeError: Can only use .dt accessor with datetimelike values EDITED: Date columns are like this: 0 2014-01-01 1 2014-01-01 2 2014-01-01 3 2014-01-01 4 2014-01-03 5 2014-01-03 6 2014-01-03 7 2014-01-07 8 2014-01-08 9 2014-01-09 Do you have any ideas? Thank you very much!

Output: Your problem here is that to_datetime silently failed so the dtype remained as str/object, if you set param errors='coerce' then if the conversion fails for any particular string then those rows are set to NaT. df['Date'] = pd.to_datetime(df['Date'], errors='coerce') So you need to find out what is wrong with those specific row values. See the docs


Instruction:
"OSError: [Errno 17] File exists" when trying to use os.makedirs

I have several threads running in parallel from Python on a cluster system. Each python thread outputs to a directory mydir. Each script, before outputting checks if mydir exists and if not creates it: if not os.path.isdir(mydir): os.makedirs(mydir) but this yields the error: os.makedirs(self.log_dir) File "/usr/lib/python2.6/os.py", line 157, in makedirs mkdir(name,mode) OSError: [Errno 17] File exists I suspect it might be due to a race condition, where one job creates the dir before the other gets to it. Is this possible? If so, how can this error be avoided? I'm not sure it's a race condition so was wondering if other issues in Python can cause this odd error.

Output: Any time code can execute between when you check something and when you act on it, you will have a race condition. One way to avoid this (and the usual way in Python) is to just try and then handle the exception while True: mydir = next_dir_name() try: os.makedirs(mydir) break except OSError, e: if e.errno != errno.EEXIST: raise # time.sleep might help here pass If you have a lot of threads trying to make a predictable series of directories this will still raise a lot of exceptions, but you will get there in the end. Better to just have one thread creating the dirs in that case


Instruction:
How to define default value if empty user input in Python?

Here I have to set the default value if the user will enter the value from the keyboard. Here is the code that user can enter value: input = int(raw_input("Enter the inputs : ")) Here the value will be assigned to a variable input after entering the value and hitting Enter. Is there any method that if we don't enter the value and directly hit the Enter key, the variable will be directly assigned to a default value, say as input = 0.025?

Output: Python 3: inp = int(input('Enter the inputs: ').strip() or "42") Python 2: inp = int(raw_input('Enter the inputs: ').strip() or "42") How does it work? If nothing was entered then input/raw_input returns empty string. Empty string in Python is False, bool("") -> False. Operator or returns first truthy value, which in this case is "42". This is not sophisticated input validation. It's slightly more robust with the addition of .strip(). That handles the possibility of the user entering spaces or other whitespace, which then would be True.


Instruction:
how to use python2.7 pip instead of default pip

I just installed python 2.7 and also pip to the 2.7 site package. When I get the version with: pip -V It shows: pip 1.3.1 from /usr/lib/python2.6/site-packages (python 2.6) How do I use the 2.7 version of pip located at: /usr/local/lib/python2.7/site-packages

Output: There should be a binary called "pip2.7" installed at some location included within your $PATH variable. You can find that out by typing which pip2.7 This should print something like '/usr/local/bin/pip2.7' to your stdout. If it does not print anything like this, it is not installed. In that case, install it by running $ wget https://bootstrap.pypa.io/pip/2.7/get-pip.py $ sudo python2.7 get-pip.py Now, you should be all set, and which pip2.7 should return the correct output.


Instruction:
Python: download a file from an FTP server

I'm trying to download some public data files. I screenscrape to get the links to the files, which all look something like this: ftp://ftp.cdc.gov/pub/Health_Statistics/NCHS/nhanes/2001-2002/L28POC_B.xpt I can't find any documentation on the Requests library website.

Output: The requests library doesn't support ftp:// links. To download a file from an FTP server you could use urlretrieve: import urllib.request urllib.request.urlretrieve('ftp://server/path/to/file', 'file') # if you need to pass credentials: # urllib.request.urlretrieve('ftp://username:password@server/path/to/file', 'file') Or urlopen: import shutil import urllib.request from contextlib import closing with closing(urllib.request.urlopen('ftp://server/path/to/file')) as r: with open('file', 'wb') as f: shutil.copyfileobj(r, f) Python 2: import shutil import urllib2 from contextlib import closing with closing(urllib2.urlopen('ftp://server/path/to/file')) as r: with open('file', 'wb') as f: shutil.copyfileobj(r, f)


Instruction:
How do I run Python asyncio code in a Jupyter notebook?

I have some asyncio code which runs fine in the Python interpreter (CPython 3.6.2). I would now like to run this inside a Jupyter notebook with an IPython kernel. I can run it with import asyncio asyncio.get_event_loop().run_forever() and while that seems to work it also seems to block the notebook and doesn't seem to play nice with the notebook. My understanding is that Jupyter uses Tornado under the hood so I tried to install a Tornado event loop as recommended in the Tornado docs: from tornado.platform.asyncio import AsyncIOMainLoop AsyncIOMainLoop().install() However that gives the following error: --------------------------------------------------------------------------- AssertionError Traceback (most recent call last) <ipython-input-1-1139449343fc> in <module>() 1 from tornado.platform.asyncio import AsyncIOMainLoop ----> 2 AsyncIOMainLoop().install() ~\AppData\Local\Continuum\Anaconda3\envs\numismatic\lib\site- packages\tornado\ioloop.py in install(self) 179 `IOLoop` (e.g., :class:`tornado.httpclient.AsyncHTTPClient`). 180 """ --> 181 assert not IOLoop.initialized() 182 IOLoop._instance = self 183 AssertionError: Finally I found the following page: http://ipywidgets.readthedocs.io/en/stable/examples/Widget%20Asynchronous.html so I added a cell with the following code: import asyncio from ipykernel.eventloops import register_integration @register_integration('asyncio') def loop_asyncio(kernel): '''Start a kernel with asyncio event loop support.''' loop = asyncio.get_event_loop() def kernel_handler(): loop.call_soon(kernel.do_one_iteration) loop.call_later(kernel._poll_interval, kernel_handler) loop.call_soon(kernel_handler) try: if not loop.is_running(): loop.run_forever() finally: loop.run_until_complete(loop.shutdown_asyncgens()) loop.close() and in the next cell I ran: %gui asyncio That worked but I don't really understand why and how it works. Can someone please explain that to me?

Output: EDIT FEB 21st, 2019: Problem Fixed This is no longer an issue on the latest version of Jupyter Notebook. Authors of Jupyter Notebook detailed the case here. You may now just do: import asyncio async def func(): print("started.") await asyncio.sleep(2) print("done.") await func() Answer below was the original response that was marked correct by the op. This was posted quite a bit ago, but in case other people are looking for an explanation and solution to the problem of running asynchronous code inside Jupyter Notebook; Jupyter's Tornado 5.0 update bricked asyncio functionalities after the addition of its own asyncio event loop: Thus, for any asyncio functionality to run on Jupyter Notebook you cannot invoke a loop.run_until_complete(...), since the loop you will receive from asyncio.get_event_loop() will be active. Instead, you must either add the task to the current event loop: import asyncio loop = asyncio.get_event_loop() loop.create_task(some_async_function()) Or get the results via run_coroutine_threadsafe: import asyncio loop = asyncio.get_event_loop() asyncio.run_coroutine_threadsafe(some_async_function(), loop)


Instruction:
How to test if a list contains another list as a contiguous subsequence?

How can I test if a list contains another list (ie. it's a contiguous subsequence). Say there was a function called contains: contains([1,2], [-1, 0, 1, 2]) # Returns [2, 3] (contains returns [start, end]) contains([1,3], [-1, 0, 1, 2]) # Returns False contains([1, 2], [[1, 2], 3]) # Returns False contains([[1, 2]], [[1, 2], 3]) # Returns [0, 0] Edit: contains([2, 1], [-1, 0, 1, 2]) # Returns False contains([-1, 1, 2], [-1, 0, 1, 2]) # Returns False contains([0, 1, 2], [-1, 0, 1, 2]) # Returns [1, 3]

Output: Here is my version: def contains(small, big): for i in xrange(len(big)-len(small)+1): for j in xrange(len(small)): if big[i+j] != small[j]: break else: return i, i+len(small) return False It returns a tuple of (start, end+1) since I think that is more pythonic, as Andrew Jaffe points out in his comment. It does not slice any sublists so should be reasonably efficient. One point of interest for newbies is that it uses the else clause on the for statement - this is not something I use very often but can be invaluable in situations like this. This is identical to finding substrings in a string, so for large lists it may be more efficient to implement something like the Boyer-Moore algorithm. Note: If you are using Python3, change xrange to range.


Instruction:
Python Decimals format

What is a good way to format a python decimal like this way? 1.00 --> '1' 1.20 --> '1.2' 1.23 --> '1.23' 1.234 --> '1.23' 1.2345 --> '1.23'

Output: If you have Python 2.6 or newer, use format: '{0:.3g}'.format(num) For Python 2.5 or older: '%.3g'%(num) Explanation: {0}tells format to print the first argument -- in this case, num. Everything after the colon (:) specifies the format_spec. .3 sets the precision to 3. g removes insignificant zeros. See http://en.wikipedia.org/wiki/Printf#fprintf For example: tests=[(1.00, '1'), (1.2, '1.2'), (1.23, '1.23'), (1.234, '1.23'), (1.2345, '1.23')] for num, answer in tests: result = '{0:.3g}'.format(num) if result != answer: print('Error: {0} --> {1} != {2}'.format(num, result, answer)) exit() else: print('{0} --> {1}'.format(num,result)) yields 1.0 --> 1 1.2 --> 1.2 1.23 --> 1.23 1.234 --> 1.23 1.2345 --> 1.23 Using Python 3.6 or newer, you could use f-strings: In [40]: num = 1.234; f'{num:.3g}' Out[40]: '1.23'


Instruction:
Drop all data in a pandas dataframe

I would like to drop all data in a pandas dataframe, but am getting TypeError: drop() takes at least 2 arguments (3 given). I essentially want a blank dataframe with just my columns headers. import pandas as pd web_stats = {'Day': [1, 2, 3, 4, 2, 6], 'Visitors': [43, 43, 34, 23, 43, 23], 'Bounce_Rate': [3, 2, 4, 3, 5, 5]} df = pd.DataFrame(web_stats) df.drop(axis=0, inplace=True) print df

Output: You need to pass the labels to be dropped. df.drop(df.index, inplace=True) By default, it operates on axis=0. You can achieve the same with df.iloc[:0] which is much more efficient.


Instruction:
Fill a new pandas column with row numbers

I have the following DataFrame data with random index values: A B 100 0 7 203 5 4 5992 0 10 2003 9 8 20 10 5 12 6 2 I would like to add a new column 'C' with row numbers. For example: A B C 100 0 7 0 203 5 4 1 5992 0 10 2 2003 9 8 3 20 10 5 4 12 6 2 5

Output: Use numpy.arange or range by length of DataFrame: df['C'] = np.arange(len(df)) df['C'] = range(len(df)) Or you can use DataFrame.shape, thank you @Mehmet Burak Sayıcı: df['C'] = np.arange(df.shape[0]) print (df) A B C 100 0 7 0 203 5 4 1 5992 0 10 2 2003 9 8 3 20 10 5 4 12 6 2 5


Instruction:
How to multiply matrices in PyTorch?

With numpy, I can do a simple matrix multiplication like this: a = numpy.ones((3, 2)) b = numpy.ones((2, 1)) result = a.dot(b) However, this does not work with PyTorch: a = torch.ones((3, 2)) b = torch.ones((2, 1)) result = torch.dot(a, b) This code throws the following error: RuntimeError: 1D tensors expected, but got 2D and 2D tensors How do I perform matrix multiplication in PyTorch?

Output: Use torch.mm: torch.mm(a, b) torch.dot() behaves differently to np.dot(). There's been some discussion about what would be desirable here. Specifically, torch.dot() treats both a and b as 1D vectors (irrespective of their original shape) and computes their inner product. The error is thrown because this behaviour makes your a a vector of length 6 and your b a vector of length 2; hence their inner product can't be computed. For matrix multiplication in PyTorch, use torch.mm(). Numpy's np.dot() in contrast is more flexible; it computes the inner product for 1D arrays and performs matrix multiplication for 2D arrays. torch.matmul performs matrix multiplications if both arguments are 2D and computes their dot product if both arguments are 1D. For inputs of such dimensions, its behaviour is the same as np.dot. It also lets you do broadcasting or matrix x matrix, matrix x vector and vector x vector operations in batches. # 1D inputs, same as torch.dot a = torch.rand(n) b = torch.rand(n) torch.matmul(a, b) # torch.Size([]) # 2D inputs, same as torch.mm a = torch.rand(m, k) b = torch.rand(k, j) torch.matmul(a, b) # torch.Size([m, j])


Instruction:
How to perform element-wise Boolean operations on NumPy arrays

For example, I would like to create a mask that masks elements with value between 40 and 60: foo = np.asanyarray(range(100)) mask = (foo < 40).__or__(foo > 60) Which just looks ugly. I can't write (foo < 40) or (foo > 60) because I end up with: ValueError Traceback (most recent call last) ... ----> 1 (foo < 40) or (foo > 60) ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() Is there a canonical way of doing element-wise Boolean operations on NumPy arrays with good looking code?

Output: Try this: mask = (foo < 40) | (foo > 60) Note: the __or__ method in an object overloads the bitwise or operator (|), not the Boolean or operator.


Instruction:
python list comprehension double for

vec = [[1,2,3], [4,5,6], [7,8,9]] print [num for elem in vec for num in elem] <----- this >>> [1, 2, 3, 4, 5, 6, 7, 8, 9] This is tricking me out. I understand elem is the lists inside of the list from for elem in vic I don't quite understand the usage of num and for num in elem in the beginning and the end. How does python interpret this? What's the order it looks at?

Output: Lets break it down. A simple list-comprehension: [x for x in collection] This is easy to understand if we break it into parts: [A for B in C] A is the item that will be in the resulting list B is each item in the collection C C is the collection itself. In this way, one could write: [x.lower() for x in words] In order to convert all words in a list to lowercase. It is when we complicate this with another list like so: [x for y in collection for x in y] # [A for B in C for D in E] Here, something special happens. We want our final list to include A items, and A items are found inside B items, so we have to tell the list-comprehension that. A is the item that will be in the resulting list B is each item in the collection C C is the collection itself D is each item in the collection E (in this case, also A) E is another collection (in this case, B) This logic is similar to the normal for loop: for y in collection: # for B in C: for x in y: # for D in E: (in this case: for A in B) # receive x # # receive A To expand on this, and give a great example + explanation, imagine that there is a train. The train engine (the front) is always going to be there (the result of the list-comprehension) Then, there are any number of train cars, each train car is in the form: for x in y A list comprehension could look like this: [z for b in a for c in b for d in c ... for z in y] Which would be like having this regular for-loop: for b in a: for c in b: for d in c: ... for z in y: # have z In other words, instead of going down a line and indenting, in a list-comprehension you just add the next loop on to the end. To go back to the train analogy: Engine - Car - Car - Car ... Tail What is the tail? The tail is a special thing in list-comprehensions. You don't need one, but if you have a tail, the tail is a condition, look at this example: [line for line in file if not line.startswith('#')] This would give you every line in a file as long as the line didn't start with a hash character (#), others are just skipped. The trick to using the "tail" of the train is that it is checked for True/False at the same time as you have your final 'Engine' or 'result' from all the loops, the above example in a regular for-loop would look like this: for line in file: if not line.startswith('#'): # have line please note: Though in my analogy of a train there is only a 'tail' at the end of the train, the condition or 'tail' can be after every 'car' or loop... for example: >>> z = [[1,2,3,4],[5,6,7,8],[9,10,11,12]] >>> [x for y in z if sum(y)>10 for x in y if x < 10] [5, 6, 7, 8, 9] In regular for-loop: >>> for y in z: if sum(y)>10: for x in y: if x < 10: print x 5 6 7 8 9


Instruction:
How to link PyCharm with PySpark?

I'm new with apache spark and apparently I installed apache-spark with homebrew in my macbook: Last login: Fri Jan 8 12:52:04 on console user@MacBook-Pro-de-User-2:~$ pyspark Python 2.7.10 (default, Jul 13 2015, 12:05:58) [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin Type "help", "copyright", "credits" or "license" for more information. Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 16/01/08 14:46:44 INFO SparkContext: Running Spark version 1.5.1 16/01/08 14:46:46 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/01/08 14:46:47 INFO SecurityManager: Changing view acls to: user 16/01/08 14:46:47 INFO SecurityManager: Changing modify acls to: user 16/01/08 14:46:47 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(user); users with modify permissions: Set(user) 16/01/08 14:46:50 INFO Slf4jLogger: Slf4jLogger started 16/01/08 14:46:50 INFO Remoting: Starting remoting 16/01/08 14:46:51 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.1.64:50199] 16/01/08 14:46:51 INFO Utils: Successfully started service 'sparkDriver' on port 50199. 16/01/08 14:46:51 INFO SparkEnv: Registering MapOutputTracker 16/01/08 14:46:51 INFO SparkEnv: Registering BlockManagerMaster 16/01/08 14:46:51 INFO DiskBlockManager: Created local directory at /private/var/folders/5x/k7n54drn1csc7w0j7vchjnmc0000gn/T/blockmgr-769e6f91-f0e7-49f9-b45d-1b6382637c95 16/01/08 14:46:51 INFO MemoryStore: MemoryStore started with capacity 530.0 MB 16/01/08 14:46:52 INFO HttpFileServer: HTTP File server directory is /private/var/folders/5x/k7n54drn1csc7w0j7vchjnmc0000gn/T/spark-8e4749ea-9ae7-4137-a0e1-52e410a8e4c5/httpd-1adcd424-c8e9-4e54-a45a-a735ade00393 16/01/08 14:46:52 INFO HttpServer: Starting HTTP Server 16/01/08 14:46:52 INFO Utils: Successfully started service 'HTTP file server' on port 50200. 16/01/08 14:46:52 INFO SparkEnv: Registering OutputCommitCoordinator 16/01/08 14:46:52 INFO Utils: Successfully started service 'SparkUI' on port 4040. 16/01/08 14:46:52 INFO SparkUI: Started SparkUI at http://192.168.1.64:4040 16/01/08 14:46:53 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set. 16/01/08 14:46:53 INFO Executor: Starting executor ID driver on host localhost 16/01/08 14:46:53 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 50201. 16/01/08 14:46:53 INFO NettyBlockTransferService: Server created on 50201 16/01/08 14:46:53 INFO BlockManagerMaster: Trying to register BlockManager 16/01/08 14:46:53 INFO BlockManagerMasterEndpoint: Registering block manager localhost:50201 with 530.0 MB RAM, BlockManagerId(driver, localhost, 50201) 16/01/08 14:46:53 INFO BlockManagerMaster: Registered BlockManager Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/\_\ version 1.5.1 /_/ Using Python version 2.7.10 (default, Jul 13 2015 12:05:58) SparkContext available as sc, HiveContext available as sqlContext. >>> I would like start playing in order to learn more about MLlib. However, I use Pycharm to write scripts in python. The problem is: when I go to Pycharm and try to call pyspark, Pycharm can not found the module. I tried adding the path to Pycharm as follows: Then from a blog I tried this: import os import sys # Path for spark source folder os.environ['SPARK_HOME']="/Users/user/Apps/spark-1.5.2-bin-hadoop2.4" # Append pyspark to Python Path sys.path.append("/Users/user/Apps/spark-1.5.2-bin-hadoop2.4/python/pyspark") try: from pyspark import SparkContext from pyspark import SparkConf print ("Successfully imported Spark Modules") except ImportError as e: print ("Can not import Spark Modules", e) sys.exit(1) And still can not start using PySpark with Pycharm, any idea of how to "link" PyCharm with apache-pyspark?. Update: Then I search for apache-spark and python path in order to set the environment variables of Pycharm: apache-spark path: user@MacBook-Pro-User-2:~$ brew info apache-spark apache-spark: stable 1.6.0, HEAD Engine for large-scale data processing https://spark.apache.org/ /usr/local/Cellar/apache-spark/1.5.1 (649 files, 302.9M) * Poured from bottle From: https://github.com/Homebrew/homebrew/blob/master/Library/Formula/apache-spark.rb python path: user@MacBook-Pro-User-2:~$ brew info python python: stable 2.7.11 (bottled), HEAD Interpreted, interactive, object-oriented programming language https://www.python.org /usr/local/Cellar/python/2.7.10_2 (4,965 files, 66.9M) * Then with the above information I tried to set the environment variables as follows: Any idea of how to correctly link Pycharm with pyspark? Then when I run a python script with the above configuration I have this exception: /usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/bin/python2.7 /Users/user/PycharmProjects/spark_examples/test_1.py Traceback (most recent call last): File "/Users/user/PycharmProjects/spark_examples/test_1.py", line 1, in <module> from pyspark import SparkContext ImportError: No module named pyspark UPDATE: Then I tried this configurations proposed by @zero323 Configuration 1: /usr/local/Cellar/apache-spark/1.5.1/ out: user@MacBook-Pro-de-User-2:/usr/local/Cellar/apache-spark/1.5.1$ ls CHANGES.txt NOTICE libexec/ INSTALL_RECEIPT.json README.md LICENSE bin/ Configuration 2: /usr/local/Cellar/apache-spark/1.5.1/libexec out: user@MacBook-Pro-de-User-2:/usr/local/Cellar/apache-spark/1.5.1/libexec$ ls R/ bin/ data/ examples/ python/ RELEASE conf/ ec2/ lib/ sbin/

Output: With PySpark package (Spark 2.2.0 and later) With SPARK-1267 being merged you should be able to simplify the process by pip installing Spark in the environment you use for PyCharm development. Go to File -> Settings -> Project Interpreter Click on install button and search for PySpark Click on install package button. Manually with user provided Spark installation Create Run configuration: Go to Run -> Edit configurations Add new Python configuration Set Script path so it points to the script you want to execute Edit Environment variables field so it contains at least: SPARK_HOME - it should point to the directory with Spark installation. It should contain directories such as bin (with spark-submit, spark-shell, etc.) and conf (with spark-defaults.conf, spark-env.sh, etc.) PYTHONPATH - it should contain $SPARK_HOME/python and optionally $SPARK_HOME/python/lib/py4j-some-version.src.zip if not available otherwise. some-version should match Py4J version used by a given Spark installation (0.8.2.1 - 1.5, 0.9 - 1.6, 0.10.3 - 2.0, 0.10.4 - 2.1, 0.10.4 - 2.2, 0.10.6 - 2.3, 0.10.7 - 2.4) Apply the settings Add PySpark library to the interpreter path (required for code completion): Go to File -> Settings -> Project Interpreter Open settings for an interpreter you want to use with Spark Edit interpreter paths so it contains path to $SPARK_HOME/python (an Py4J if required) Save the settings Optionally Install or add to path type annotations matching installed Spark version to get better completion and static error detection (Disclaimer - I am an author of the project). Finally Use newly created configuration to run your script.


Instruction:
re.sub replace with matched content

Trying to get to grips with regular expressions in Python, I'm trying to output some HTML highlighted in part of a URL. My input is images/:id/size my output should be images/:id/size If I do this in Javascript method = 'images/:id/size'; method = method.replace(/\:([a-z]+)/, '$1') alert(method) I get the desired result, but if I do this in Python >>> method = 'images/:id/huge' >>> re.sub('\:([a-z]+)', '$1', method) 'images/$1/huge' I don't, how do I get Python to return the correct result rather than $1? Is re.sub even the right function to do this?

Output: Simply use \1 instead of $1: In [1]: import re In [2]: method = 'images/:id/huge' In [3]: re.sub(r'(:[a-z]+)', r'\1', method) Out[3]: 'images/:id/huge' Also note the use of raw strings (r'...') for regular expressions. It is not mandatory but removes the need to escape backslashes, arguably making the code slightly more readable.


Instruction:
Convert from '_io.BytesIO' to a bytes-like object in python3.6?

I am using this function to uncompress the body of a HTTP response if it is compressed with gzip, compress or deflate. def uncompress_body(self, compression_type, body): if compression_type == 'gzip' or compression_type == 'compress': return zlib.decompress(body) elif compression_type == 'deflate': compressor = zlib.compressobj(9, zlib.DEFLATED, -zlib.MAX_WBITS) compressed = compressor.compress(body) compressed += compressor.flush() return base64.b64encode(compressed) return body However python throws this error message. TypeError: a bytes-like object is required, not '_io.BytesIO' on this line: return zlib.decompress(body) Essentially, how do I convert from '_io.BytesIO' to a bytes-like object?

Output: It's a file-like object. Read them: >>> b = io.BytesIO(b'hello') >>> b.read() b'hello' If the data coming in from body is too large to read into memory, you'll want to refactor your code and use zlib.decompressobj instead of zlib.decompress.


Instruction:
How can I make multiple replacements in a string using a dictionary?

Suppose we have: d = { 'Спорт':'Досуг', 'russianA':'englishA' } s = 'Спорт russianA' How can I replace each appearance within s of any of d's keys, with the corresponding value (in this case, the result would be 'Досуг englishA')?

Output: Using re: import re s = 'Спорт not russianA' d = { 'Спорт':'Досуг', 'russianA':'englishA' } keys = (re.escape(k) for k in d.keys()) pattern = re.compile(r'\b(' + '|'.join(keys) + r')\b') result = pattern.sub(lambda x: d[x.group()], s) # Output: 'Досуг not englishA' This will match whole words only. If you don't need that, use the pattern: pattern = re.compile('|'.join(re.escape(k) for k in d.keys())) Note that in this case you should sort the words descending by length if some of your dictionary entries are substrings of others.


Instruction:
TypeError: only length-1 arrays can be converted to Python scalars while plot showing

I have such Python code: import numpy as np import matplotlib.pyplot as plt def f(x): return np.int(x) x = np.arange(1, 15.1, 0.1) plt.plot(x, f(x)) plt.show() And such error: TypeError: only length-1 arrays can be converted to Python scalars How can I fix it?

Output: The error "only length-1 arrays can be converted to Python scalars" is raised when the function expects a single value but you pass an array instead. np.int was an alias for the built-in int, which is deprecated in numpy v1.20. The argument for int should be a scalar and it does not accept array-like objects. In general, if you want to apply a function to each element of the array, you can use np.vectorize: import numpy as np import matplotlib.pyplot as plt def f(x): return int(x) f2 = np.vectorize(f) x = np.arange(1, 15.1, 0.1) plt.plot(x, f2(x)) plt.show() You can skip the definition of f(x) and just pass the function int to the vectorize function: f2 = np.vectorize(int). Note that np.vectorize is just a convenience function and basically a for loop. That will be inefficient over large arrays. Whenever you have the possibility, use truly vectorized functions or methods (like astype(int) as @FFT suggests).


Instruction:
Django - How to make a variable available to all templates?

I would like to know how to pass a variable to all my templates, without repeating the same code on every method in my views.py file? In the example below I would like to make categories (an array of category objects) available to all templates in the web app. Eg: I would like to avoid writing 'categories':categories on every method. Is it possible? One view method def front_page(request): categories = Category.objects.all() if is_logged_in(request) is False: return render_to_response('users/signup.html', {'is_logged_in': is_logged_in(request), 'categories':categories}, context_instance=RequestContext(request)) else: return render_to_response('users/front_page.html', {'is_logged_in': is_logged_in(request), 'categories':categories},context_instance=RequestContext(request)) Another view method def another_view_method(request): categories = Category.objects.all() return render_to_response('eg/front_page.html', {'is_logged_in': is_logged_in(request), 'categories':categories},context_instance=RequestContext(request))

Output: What you want is a context processor, and it's very easy to create one. Assuming you have an app named custom_app, follow the next steps: Add custom_app to INSTALLED_APPS in settings.py (you've done it already, right?); Create a file named context_processors.py into custom_app folder; Add the following code to that new file: def categories_processor(request): categories = Category.objects.all() return {'categories': categories} Add context_processors.py to TEMPLATE_CONTEXT_PROCESSORS in settings.py TEMPLATE_CONTEXT_PROCESSORS += ("custom_app.context_processors.categories_processor", ) And now you can use {{categories}} in all the templates :D As of Django 1.8 To add a TEMPLATE_CONTEXT_PROCESSORS, in the settings you must add the next code: TEMPLATES[0]['OPTIONS']['context_processors'].append("custom_app.context_processors.categories_processor") Or include that string directly in the OPTIONS.context_processors key in your TEMPLATES setting.


Instruction:
Handle Flask requests concurrently with threaded=True

What exactly does passing threaded = True to app.run() do? My application processes input from the user, and takes a bit of time to do so. During this time, the application is unable to handle other requests. I have tested my application with threaded=True and it allows me to handle multiple requests concurrently.

Output: As of Flask 1.0, the WSGI server included with Flask is run in threaded mode by default. Prior to 1.0, or if you disable threading, the server is run in single-threaded mode, and can only handle one request at a time. Any parallel requests will have to wait until they can be handled, which can lead to issues if you tried to contact your own server from a request. With threaded=True requests are each handled in a new thread. How many threads your server can handle concurrently depends entirely on your OS and what limits it sets on the number of threads per process. The implementation uses the SocketServer.ThreadingMixIn class, which sets no limits to the number of threads it can spin up. Note that the Flask server is designed for development only. It is not a production-ready server. Don't rely on it to run your site on the wider web. Use a proper WSGI server (like gunicorn or uWSGI) instead.


Instruction:
Why does Python start at index -1 (as opposed to 0) when indexing a list from the end?

list = ["a", "b", "c", "d"] print(list[3]) # Number 3 is "d" print(list[-4]) # Number -4 is "a"

Output: To explain it in another way, because -0 is equal to 0, if backward starts from 0, it is ambiguous to the interpreter. If you are confused about -, and looking for another way to index backwards more understandably, you can try ~, it is a mirror of forward: arr = ["a", "b", "c", "d"] print(arr[~0]) # d print(arr[~1]) # c The typical usages for ~ are like "swap mirror node" or "find median in a sort list": """swap mirror node""" def reverse(arr: List[int]) -> None: for i in range(len(arr) // 2): arr[i], arr[~i] = arr[~i], arr[i] """find median in a sort list""" def median(arr: List[float]) -> float: mid = len(arr) // 2 return (arr[mid] + arr[~mid]) / 2 """deal with mirror pairs""" # verify the number is strobogrammatic, strobogrammatic number looks the same when rotated 180 degrees def is_strobogrammatic(num: str) -> bool: return all(num[i] + num[~i] in '696 00 11 88' for i in range(len(num) // 2 + 1)) ~ actually is a math trick of inverse code and complement code, and it is more easy to understand in some situations. Discussion about whether should use python tricks like ~: In my opinion, if it is a code maintained by yourself, you can use any trick to avoid potential bug or achieve goal easier, because of maybe a high readability and usability. But in team work, avoid using 'too clever' code, may bring troubles to your co-workers. For example, here is one concise code from Stefan Pochmann to solve this problem. I learned a lot from his code. But some are just for fun, too hackish to use. # a strobogrammatic number is a number that looks the same when rotated 180 degrees (looked at upside down) # find all strobogrammatic numbers that are of length = n def findStrobogrammatic(self, n): nums = n % 2 * list('018') or [''] while n > 1: n -= 2 # n < 2 is so genius here nums = [a + num + b for a, b in '00 11 88 69 96'.split()[n < 2:] for num in nums] return nums I have summarized python tricks like this, in case you are interested.


Instruction:
Playing mp3 song on python

I want to play my song (mp3) from python, can you give me a simplest command to do that? This is not correct: import wave w = wave.open("e:/LOCAL/Betrayer/Metalik Klinik1-Anak Sekolah.mp3","r")

Output: Try this. It's simplistic, but probably not the best method. from pygame import mixer # Load the popular external library mixer.init() mixer.music.load('e:/LOCAL/Betrayer/Metalik Klinik1-Anak Sekolah.mp3') mixer.music.play() Please note that pygame's support for MP3 is limited. Also, as pointed out by Samy Bencherif, there won't be any silly pygame window popup when you run the above code. Installation is simple - $pip install pygame Update: Above code will only play the music if ran interactively, since the play() call will execute instantaneously and the script will exit. To avoid this, you could instead use the following to wait for the music to finish playing and then exit the program, when running the code as a script. import time from pygame import mixer mixer.init() mixer.music.load("/file/path/mymusic.ogg") mixer.music.play() while mixer.music.get_busy(): # wait for music to finish playing time.sleep(1)


Instruction:
How to convert an H:MM:SS time string to seconds in Python?

Basically I have the inverse of this problem: Python Time Seconds to h:m:s I have a string in the format H:MM:SS (always 2 digits for minutes and seconds), and I need the integer number of seconds that it represents. How can I do this in python? For example: "1:23:45" would produce an output of 5025 "0:04:15" would produce an output of 255 "0:00:25" would produce an output of 25 etc

Output: def get_sec(time_str): """Get seconds from time.""" h, m, s = time_str.split(':') return int(h) * 3600 + int(m) * 60 + int(s) print(get_sec('1:23:45')) print(get_sec('0:04:15')) print(get_sec('0:00:25'))


Instruction:
How to upload a file to Google Cloud Storage on Python 3?

How can I upload a file to Google Cloud Storage from Python 3? Eventually Python 2, if it's infeasible from Python 3. I've looked and looked, but haven't found a solution that actually works. I tried boto, but when I try to generate the necessary .boto file through gsutil config -e, it keeps saying that I need to configure authentication through gcloud auth login. However, I have done the latter a number of times, without it helping.

Output: Use the standard gcloud library, which supports both Python 2 and Python 3. Example of Uploading File to Cloud Storage from gcloud import storage from oauth2client.service_account import ServiceAccountCredentials import os credentials_dict = { 'type': 'service_account', 'client_id': os.environ['BACKUP_CLIENT_ID'], 'client_email': os.environ['BACKUP_CLIENT_EMAIL'], 'private_key_id': os.environ['BACKUP_PRIVATE_KEY_ID'], 'private_key': os.environ['BACKUP_PRIVATE_KEY'], } credentials = ServiceAccountCredentials.from_json_keyfile_dict( credentials_dict ) client = storage.Client(credentials=credentials, project='myproject') bucket = client.get_bucket('mybucket') blob = bucket.blob('myfile') blob.upload_from_filename('myfile')


Instruction:
Running Jupyter notebook in a virtualenv: installed sklearn module not available

I have installed a created a virtualenv machinelearn and installed a few python modules (pandas, scipy and sklearn) in that environment. When I run jupyter notebook, I can import pandas and scipy in my notebooks - however, when I try to import sklearn, I get the following error message: import sklearn --------------------------------------------------------------------------- ImportError Traceback (most recent call last) <ipython-input-1-8fd979e02004> in <module>() ----> 1 import sklearn ImportError: No module named 'sklearn' I am able to import all modules, at the command line - so I know they have been successfully installed: (machinelearn) me@yourbox:~/path/to/machinelearn$ python -c "import pandas, scipy, sklearn" (machinelearn) me@yourbox:~/path/to/machinelearn$ How can I import sklearn in my jupyter notebook running in a virtualenv?

Output: You probably have not installed jupyter / IPython in your virtualenv. Try the following: python -c "import IPython" and check that the jupyter command found in your $PATH is the one from the bin folder of your venv: which jupyter For windows users in a powershell console, you can use the following to check that the jupyter command in your $env:Path is the one from the Scripts folder of you venv: get-command jupyter Edit: if this is the problem, just run python -m pip install jupyter in your venv. Edit 2: actually you might also need: python -m ipykernel install --user --name=my-virtualenv-name and then switch the kernel named "my-virtualenv-name" in the jupyter user interface. Edit 3: maybe the --user flag in the last command is a bad idea: python -m ipykernel install --name=my-virtualenv-name


Instruction:
Python: How to use RegEx in an if statement?

I have the following code which looks through the files in one directory and copies files that contain a certain string into another directory, but I am trying to use Regular Expressions as the string could be upper and lowercase or a mix of both. Here is the code that works, before I tried to use RegEx's import os import re import shutil def test(): os.chdir("C:/Users/David/Desktop/Test/MyFiles") files = os.listdir(".") os.mkdir("C:/Users/David/Desktop/Test/MyFiles2") for x in (files): inputFile = open((x), "r") content = inputFile.read() inputFile.close() if ("Hello World" in content) shutil.copy(x, "C:/Users/David/Desktop/Test/MyFiles2") Here is my code when I have tried to use RegEx's import os import re import shutil def test2(): os.chdir("C:/Users/David/Desktop/Test/MyFiles") files = os.listdir(".") os.mkdir("C:/Users/David/Desktop/Test/MyFiles2") regex_txt = "facebook.com" for x in (files): inputFile = open((x), "r") content = inputFile.read() inputFile.close() regex = re.compile(regex_txt, re.IGNORECASE) Im guessing that I need a line of code that is something like if regex = re.compile(regex_txt, re.IGNORECASE) == True But I cant seem to get anything to work, if someone could point me in the right direction it would be appreciated.

Output: import re if re.match(regex, content): blah.. You could also use re.search depending on how you want it to match. You can run this example: """ very nice interface to try regexes: https://regex101.com/ """ # %% """Simple if statement with a regex""" import re regex = r"\s*Proof.\s*" contents = ['Proof.\n', '\nProof.\n'] for content in contents: assert re.match(regex, content), f'Failed on {content=} with {regex=}' if re.match(regex, content): print(content)


Instruction:
How to check for "200 OK"?

What is the easiest way to check whether the response received from a requests post was "200 OK" or an error has occurred? I tried doing something like this: .... resp = requests.post(my_endpoint_var, headers=header_var, data=post_data_var) print(resp) if resp == "<Response [200]>": print ('OK!') else: print ('Boo!') The output on the screen is: Response [200] (including the "<" and ">") Boo! So even though I am getting a 200, my check in the if statement is somehow not matching?

Output: According to the docs, there's a status_code property on the response object. So you can do the following: if resp.status_code == 200: print ('OK!') else: print ('Boo!') As others have pointed out, a simpler check might be to use the ok property: if resp.ok: print ('OK!') else: print ('Boo!') That is, if you want to consider all 2xx response codes and not 200 explicitly. You may also want to check Peter's answer for a more pythonic way to do this.


Instruction:
How to get multiline input from the user

I want to write a program that gets multiple line input and work with it line by line. Why isn't there any function like raw_input in Python 3? input does not allow the user to put lines separated by newline (Enter). It prints back only the first line. Can it be stored in a variable or even read it to a list?

Output: In Python 3.x the raw_input() of Python 2.x has been replaced by input() function. However in both the cases you cannot input multi-line strings, for that purpose you would need to get input from the user line by line and then .join() them using \n, or you can also take various lines and concatenate them using + operator separated by \n To get multi-line input from the user you can go like: no_of_lines = 5 lines = "" for i in xrange(no_of_lines): lines+=input()+"\n" print(lines) Or lines = [] while True: line = input() if line: lines.append(line) else: break text = '\n'.join(lines)


Instruction:
How to continue a task when Fabric receives an error

When I define a task to run on several remote servers, if the task runs on server one and exits with an error, Fabric will stop and abort the task. But I want to make fabric ignore the error and run the task on the next server. How can I make it do this? For example: $ fab site1_service_gw [site1rpt1] Executing task 'site1_service_gw' [site1fep1] run: echo 'Nm123!@#' | sudo -S route [site1fep1] err: [site1fep1] err: We trust you have received the usual lecture from the local System [site1fep1] err: Administrator. It usually boils down to these three things: [site1fep1] err: [site1fep1] err: #1) Respect the privacy of others. [site1fep1] err: #2) Think before you type. [site1fep1] err: #3) With great power comes great responsibility. [site1fep1] err: root's password: [site1fep1] err: sudo: route: command not found Fatal error: run() encountered an error (return code 1) while executing 'echo 'Nm123!@#' | sudo -S route ' Aborting.

Output: From the docs: ... Fabric defaults to a “fail-fast” behavior pattern: if anything goes wrong, such as a remote program returning a nonzero return value or your fabfile’s Python code encountering an exception, execution will halt immediately. This is typically the desired behavior, but there are many exceptions to the rule, so Fabric provides env.warn_only, a Boolean setting. It defaults to False, meaning an error condition will result in the program aborting immediately. However, if env.warn_only is set to True at the time of failure – with, say, the settings context manager – Fabric will emit a warning message but continue executing. Looks like you can exercise fine-grained control over where errors are ignored by using the settings context manager, something like so: from fabric.api import settings sudo('mkdir tmp') # can't fail with settings(warn_only=True): sudo('touch tmp/test') # can fail sudo('rm tmp') # can't fail


Instruction:
Python 3 sort a dict by its values

The only methods I found work for python2 or return only list of tuples. Is it possible to sort dictionary, e.g. {"aa": 3, "bb": 4, "cc": 2, "dd": 1}, by its values? Order of sorted dictionary I want to achieve is from largest to smallest. I want results to look like this: bb 4 aa 3 cc 2 dd 1 And after sorting I want to store it into a text file.

Output: itemgetter (see other answers) is (as I know) more efficient for large dictionaries but for the common case, I believe that d.get wins. And it does not require an extra import. >>> d = {"aa": 3, "bb": 4, "cc": 2, "dd": 1} >>> for k in sorted(d, key=d.get, reverse=True): ... k, d[k] ... ('bb', 4) ('aa', 3) ('cc', 2) ('dd', 1) Note that alternatively you can set d.__getitem__ as key function which may provide a small performance boost over d.get.


Instruction:
python dictionary passed as an input to a function acts like a global in that function rather than a local

I am very confused by the behaviour below. Cases 1, 3, and 4 perform as I would expect, but case 2 does not. Why does case 2 allow the function to change the value of the dictionary entry globally, even though the dictionary is never returned by the function? A main reason I am using functions is to isolate everything in the function from the rest of the code, but this does not seem to be possible if I choose to use the same variable names inside of the function. I was under the understanding that anything explicitly defined in a function is local to that function, but this does not seem to be the case if the dictionary is defined and passed as an input to the function. Case 1 >>> def testfun1(a): ... a=2 ... >>> a=0 >>> testfun1(a) >>> a 0 Case 2 >>> def testfun2(b): ... b['test']=2 ... >>> b={} >>> testfun2(b) >>> b {'test': 2} Case 3 >>> def testfun3(): ... c=2 ... >>> c=0 >>> testfun3() >>> c 0 Case 4 (explained by this question: Global dictionaries don't need keyword global to modify them?) >>> def testfun4(): ... d['test']=10 ... >>> d={} >>> testfun4() >>> d {'test': 10}

Output: Python's "parameter evaluation strategy" acts a bit different than the languages you're probably used to. Instead of having explicit call by value and call by reference semantics, python has call by sharing. You are essentially always passing the object itself, and the object's mutability determines whether or not it can be modified. Lists and Dicts are mutable objects. Numbers, Strings, and Tuples are not. You are passing the dictionary to the function, not a copy. Thus when you modify it, you are also modifying the original copy. To avoid this, you should first copy the dictionary before calling the function, or from within the function (passing the dictionary to the dict function should do it, i.e. testfun4(dict(d)) and defining the function as def testfun4(d):).


Instruction:
What is the difference between time.perf_counter() and time.process_time()?

I am using Jupyter notebook. I'm trying to measure how long will it take to count the Avogadro's number with Python. I found that time.perf_counter() and time.process_time() modules will be useful for this kind of work. So I tried both of them, but the result was totally different. What makes this kind of difference? Here is my code. Using time.perf_counter() import time a = 10 ** 5 def AvogadroCounting(): i = 0 while i <= a: i += 1 AvogadroCounting() t_fract = time.perf_counter() #time to count fraction of avogadro's number in Seconds print(t_fract, 'secs') And my notebook gives 693920.393636181 secs. Using time.process_time() import time a = 10 ** 5 def AvogadroCounting(): i = 0 while i <= a: i += 1 AvogadroCounting() t_fract = time.process_time() #time to count fraction of avogadro's number in Seconds print(t_fract, 'secs') and this gives 2048.768273 secs.

Output: time.perf_counter() keeps going during sleep, time.process_time() does not. time.perf_counter() → float Return the value (in fractional seconds) of a performance counter, i.e. a clock with the highest available resolution to measure a short duration. It does include time elapsed during sleep and is system-wide. The reference point of the returned value is undefined, so that only the difference between the results of consecutive calls is valid. time.process_time() → float Return the value (in fractional seconds) of the sum of the system and user CPU time of the current process. It does not include time elapsed during sleep. It is process-wide by definition. The reference point of the returned value is undefined, so that only the difference between the results of consecutive calls is valid. See the official documentation import time def pc(): start = time.perf_counter() time.sleep(1) print(time.perf_counter()-start) def pt(): start = time.process_time() time.sleep(1) print(time.process_time()-start) pc() # 0.99872320449432 pt() # 0.0


Instruction:
Why does defining __getitem__ on a class make it iterable in python?

Why does defining __getitem__ on a class make it iterable? For instance if I write: class B: def __getitem__(self, k): return k cb = B() for k in cb: print k I get the output: 0 1 2 3 4 5 ... I would really expect to see an error returned from for k in cb:.

Output: If you take a look at PEP234 defining iterators, it says: An object can be iterated over with for if it implements __iter__() or __getitem__(). An object can function as an iterator if it implements next().


Instruction:
Pandas dataframe groupby plot

I have a dataframe which is structured as: Date ticker adj_close 0 2016-11-21 AAPL 111.730 1 2016-11-22 AAPL 111.800 2 2016-11-23 AAPL 111.230 3 2016-11-25 AAPL 111.790 4 2016-11-28 AAPL 111.570 ... 8 2016-11-21 ACN 119.680 9 2016-11-22 ACN 119.480 10 2016-11-23 ACN 119.820 11 2016-11-25 ACN 120.740 ... How can I plot based on the ticker the adj_close versus Date?

Output: Simple plot, you can use: df.plot(x='Date',y='adj_close') Or you can set the index to be Date beforehand, then it's easy to plot the column you want: df.set_index('Date', inplace=True) df['adj_close'].plot() If you want a chart with one series by ticker on it You need to groupby before: df.set_index('Date', inplace=True) df.groupby('ticker')['adj_close'].plot(legend=True) If you want a chart with individual subplots: grouped = df.groupby('ticker') ncols=2 nrows = int(np.ceil(grouped.ngroups/ncols)) fig, axes = plt.subplots(nrows=nrows, ncols=ncols, figsize=(12,4), sharey=True) for (key, ax) in zip(grouped.groups.keys(), axes.flatten()): grouped.get_group(key).plot(ax=ax) ax.legend() plt.show()


Instruction:
Numpy slice of arbitrary dimensions

I would like to slice a numpy array to obtain the i-th index in the last dimension. For a 3D array, this would be: slice = myarray[:, :, i] But I am writing a function where I can take an array of arbitrary dimensions, so for a 4D array I'd need myarray[:, :, :, i], and so on. Is there a way I can obtain this slice for any array without explicitly having to write the array dimensions?

Output: There is ... or Ellipsis, which does exactly this: slice = myarray[..., i] Ellipsis is the python object, if you should want to use it outside the square bracket notation.


Instruction:
List of All Tkinter Events

In Python tkinter module, <Button-1>, <Button-2> and <Button-3> are used to identify mouse button clicks for left, middle and right buttons respectively. Likewise, <KeyPress-Return> is used for the return key press. Where can I find a list of all such events, including the names for the various keyboard keys? Tcl bind manual does not have those.

Output: A general list for Bindings and Events can be found on effbot.org or in the docs provided by New Mexico Tech whereas the name of several keys are listed here in addition to the original documentation. Here's a summary of the most common events with some keypress names explained: Event Description <Button-1> Button 1 is the leftmost button, button 2 is the middle button(where available), and button 3 the rightmost button. <Button-1>, <ButtonPress-1>, and <1> are all synonyms. For mouse wheel support under Linux, use Button-4 (scroll up) and Button-5 (scroll down) <B1-Motion> The mouse is moved, with mouse button 1 being held down (use B2 for the middle button, B3 for the right button). <ButtonRelease-1> Button 1 was released. This is probably a better choice in most cases than the Button event, because if the user accidentally presses the button, they can move the mouse off the widget to avoid setting off the event. <Double-Button-1> Button 1 was double clicked. You can use Double or Triple as prefixes. <Enter> The mouse pointer entered the widget (this event doesn't mean that the user pressed the Enter key!). <Leave> The mouse pointer left the widget. <FocusIn> Keyboard focus was moved to this widget, or to a child of this widget. <FocusOut> Keyboard focus was moved from this widget to another widget. <Return> The user pressed the Enter key. For an ordinary 102-key PC-style keyboard, the special keys are Cancel (the Break key), BackSpace, Tab, Return(the Enter key), Shift_L (any Shift key), Control_L (any Control key), Alt_L (any Alt key), Pause, Caps_Lock, Escape, Prior (Page Up), Next (Page Down), End, Home, Left, Up, Right, Down, Print, Insert, Delete, F1, F2, F3, F4, F5, F6, F7, F8, F9, F10, F11, F12, Num_Lock, and Scroll_Lock. <Key> The user pressed any key. The key is provided in the char member of the event object passed to the callback (this is an empty string for special keys). a The user typed an "a". Most printable characters can be used as is. The exceptions are space (<space>) and less than (<less>). Note that 1 is a keyboard binding, while <1> is a button binding. <Shift-Up> The user pressed the Up arrow, while holding the Shift key pressed. You can use prefixes like Alt, Shift, and Control. <Configure> The widget changed size (or location, on some platforms). The new size is provided in the width and height attributes of the event object passed to the callback. <Activate> A widget is changing from being inactive to being active. This refers to changes in the state option of a widget such as a button changing from inactive (grayed out) to active. <Deactivate> A widget is changing from being active to being inactive. This refers to changes in the state option of a widget such as a radiobutton changing from active to inactive (grayed out). <Destroy> A widget is being destroyed. <Expose> This event occurs whenever at least some part of your application or widget becomes visible after having been covered up by another window. <KeyRelease> The user let up on a key. <Map> A widget is being mapped, that is, made visible in the application. This will happen, for example, when you call the widget's .grid() method. <Motion> The user moved the mouse pointer entirely within a widget. <MouseWheel> The user moved the mouse wheel up or down. At present, this binding works on Windows and MacOS, but not under Linux. <Unmap> A widget is being unmapped and is no longer visible. <Visibility> Happens when at least some part of the application window becomes visible on the screen.


Instruction:
Set the range of the y axis in Plotly

I made a line plot with Plotly. How can I set the range of Y axis to always have it in [0; 10]? layout = go.Layout( title=go.layout.Title(text="Test", xref="paper", x=0), xaxis=go.layout.XAxis( tickmode="linear", tickfont=dict(size=10), title=go.layout.xaxis.Title(font=dict(size=14, color="#7f7f7f")), ), yaxis=go.layout.YAxis( title=go.layout.yaxis.Title(text=y, font=dict(size=14, color="#7f7f7f")), ), ) data = [go.Scatter(x=x1, y=y1)]

Output: Update for newer versions When setting up a figure you can use plotly's magic underscore notation and specify layout_yaxis_range=[<from_value>, <to_value>] like this: fig = go.Figure(data=go.Scatter(x=x, y=y, mode='lines'), layout_yaxis_range=[-4,4]) Or if you've already got a figure named fig, you can use: fig.update_layout(yaxis_range=[-4,4]) And: fig.update(layout_yaxis_range = [-4,4]) Or: fig.update_yaxes(range = [-4,4]) Figure: Complete code: # imports import pandas as pd import plotly.graph_objs as go import numpy as np # data np.random.seed(4) x = np.linspace(0, 1, 50) y = np.cumsum(np.random.randn(50)) # plotly line chart fig = go.Figure(data=go.Scatter(x=x, y=y, mode='lines'), layout_yaxis_range=[-4,4]) fig.update_layout(yaxis_range=[-4,4]) fig.show() Original answer using plotly.offline, iplot and no magic underscore notation: When setting up a figure, use: layout = go.Layout(yaxis=dict(range=[fromValue, toValue]) Or if you've already got a figure named fig, you can use: fig.update_layout(yaxis=dict(range=[fromValue,toValue])) Plot: Complete code for Jupyter Notebook: # imports from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot import pandas as pd import plotly.graph_objs as go import numpy as np # setup init_notebook_mode(connected=True) # data np.random.seed(4) x = np.linspace(0, 1, 50) y = np.cumsum(np.random.randn(50)) # line trace = go.Scatter( x=x, y=y, ) # layout layout = go.Layout(yaxis=dict(range=[-4,4]) ) # Plot fig = go.Figure(data=[trace], layout=layout) iplot(fig) Some important details: With this setup, you can easily add an y axis title like this: # layout layout = go.Layout(yaxis=dict(range=[-4,4]), title='y Axis') ) It's a little more tricky if you'd like to format that title further. I find it easiest to actually add another element with title = go.layout.yaxis.Title(text='y Axis', font=dict(size=14, color='#7f7f7f'). As long as you do it the right way, you should not experience the situation in your comment above: Thanks. I tried it. But then I have 2 definitions of yaxis in the Layout: yaxis=dict(range=[0, 10]) and yaxis=go.layout.YAxis. Therefore an error appears. Take a look at this: Plot: Complete code with y-axis text formatting: # imports from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot import pandas as pd import plotly.graph_objs as go import numpy as np # setup init_notebook_mode(connected=True) # data np.random.seed(4) x = np.linspace(0, 1, 50) y = np.cumsum(np.random.randn(50)) # line trace = go.Scatter( x=x, y=y, ) # layout layout = go.Layout( yaxis=dict(range=[-4,4], title = go.layout.yaxis.Title(text='y Axis', font=dict(size=14, color='#7f7f7f'))) ) # Plot fig = go.Figure(data=[trace], layout=layout) iplot(fig)


Instruction:
How to round a numpy array?

I have a numpy array, something like below: data = np.array([ 1.60130719e-01, 9.93827160e-01, 3.63108206e-04]) and I want to round each element to two decimal places. How can I do so?

Output: Numpy provides two identical methods to do this. Either use np.round(data, 2) or np.around(data, 2) as they are equivalent. See the documentation for more information. Examples: >>> import numpy as np >>> a = np.array([0.015, 0.235, 0.112]) >>> np.round(a, 2) array([0.02, 0.24, 0.11]) >>> np.around(a, 2) array([0.02, 0.24, 0.11]) >>> np.round(a, 1) array([0. , 0.2, 0.1])


Instruction:
Cartesian product of a dictionary of lists

I'm trying to write some code to test out the Cartesian product of a bunch of input parameters. I've looked at itertools, but its product function is not exactly what I want. Is there a simple obvious way to take a dictionary with an arbitrary number of keys and an arbitrary number of elements in each value, and then yield a dictionary with the next permutation? Input: options = {"number": [1,2,3], "color": ["orange","blue"] } print list( my_product(options) ) Example output: [ {"number": 1, "color": "orange"}, {"number": 1, "color": "blue"}, {"number": 2, "color": "orange"}, {"number": 2, "color": "blue"}, {"number": 3, "color": "orange"}, {"number": 3, "color": "blue"} ]

Output: Ok, thanks @dfan for telling me I was looking in the wrong place. I've got it now: from itertools import product def my_product(inp): return (dict(zip(inp.keys(), values)) for values in product(*inp.values()) EDIT: after years more Python experience, I think a better solution is to accept kwargs rather than a dictionary of inputs; the call style is more analogous to that of the original itertools.product. Also I think writing a generator function, rather than a function that returns a generator expression, makes the code clearer. So: import itertools def product_dict(**kwargs): keys = kwargs.keys() for instance in itertools.product(*kwargs.values()): yield dict(zip(keys, instance)) and if you need to pass in a dict, list(product_dict(**mydict)). The one notable change using kwargs rather than an arbitrary input class is that it prevents the keys/values from being ordered, at least until Python 3.6.


Instruction:
Sklearn Pipeline: Get feature names after OneHotEncode In ColumnTransformer

I want to get feature names after I fit the pipeline. categorical_features = ['brand', 'category_name', 'sub_category'] categorical_transformer = Pipeline(steps=[ ('imputer', SimpleImputer(strategy='constant', fill_value='missing')), ('onehot', OneHotEncoder(handle_unknown='ignore'))]) numeric_features = ['num1', 'num2', 'num3', 'num4'] numeric_transformer = Pipeline(steps=[ ('imputer', SimpleImputer(strategy='median')), ('scaler', StandardScaler())]) preprocessor = ColumnTransformer( transformers=[ ('num', numeric_transformer, numeric_features), ('cat', categorical_transformer, categorical_features)]) Then clf = Pipeline(steps=[('preprocessor', preprocessor), ('regressor', GradientBoostingRegressor())]) After fitting with pandas dataframe, I can get feature importances from clf.steps[1][1].feature_importances_ and I tried clf.steps[0][1].get_feature_names() but I got an error AttributeError: Transformer num (type Pipeline) does not provide get_feature_names. How can I get feature names from this?

Output: You can access the feature_names using the following snippet: clf.named_steps['preprocessor'].transformers_[1][1]\ .named_steps['onehot'].get_feature_names(categorical_features) Using sklearn >= 0.21 version, we can make it even simpler: clf['preprocessor'].transformers_[1][1]\ ['onehot'].get_feature_names(categorical_features) Reproducible example: import numpy as np import pandas as pd from sklearn.impute import SimpleImputer from sklearn.preprocessing import OneHotEncoder, StandardScaler from sklearn.pipeline import Pipeline from sklearn.compose import ColumnTransformer from sklearn.linear_model import LinearRegression df = pd.DataFrame({'brand': ['aaaa', 'asdfasdf', 'sadfds', 'NaN'], 'category': ['asdf', 'asfa', 'asdfas', 'as'], 'num1': [1, 1, 0, 0], 'target': [0.2, 0.11, 1.34, 1.123]}) numeric_features = ['num1'] numeric_transformer = Pipeline(steps=[ ('imputer', SimpleImputer(strategy='median')), ('scaler', StandardScaler())]) categorical_features = ['brand', 'category'] categorical_transformer = Pipeline(steps=[ ('imputer', SimpleImputer(strategy='constant', fill_value='missing')), ('onehot', OneHotEncoder(handle_unknown='ignore'))]) preprocessor = ColumnTransformer( transformers=[ ('num', numeric_transformer, numeric_features), ('cat', categorical_transformer, categorical_features)]) clf = Pipeline(steps=[('preprocessor', preprocessor), ('regressor', LinearRegression())]) clf.fit(df.drop('target', 1), df['target']) clf.named_steps['preprocessor'].transformers_[1][1]\ .named_steps['onehot'].get_feature_names(categorical_features) # ['brand_NaN' 'brand_aaaa' 'brand_asdfasdf' 'brand_sadfds' 'category_as' # 'category_asdf' 'category_asdfas' 'category_asfa']


Instruction:
Filter a Python list by predicate

I would want to do something like: >>> lst = [1, 2, 3, 4, 5] >>> lst.find(lambda x: x % 2 == 0) 2 >>> lst.findall(lambda x: x % 2 == 0) [2, 4] Is there anything nearing such behavior in Python's standard libraries? I know it's very easy to roll-your-own here, but I'm looking for a more standard way.

Output: You can use the filter method: >>> lst = [1, 2, 3, 4, 5] >>> filter(lambda x: x % 2 == 0, lst) [2, 4] or a list comprehension: >>> lst = [1, 2, 3, 4, 5] >>> [x for x in lst if x %2 == 0] [2, 4] to find a single element, you could try: >>> next(x for x in lst if x % 2 == 0) 2 Though that would throw an exception if nothing matches, so you'd probably want to wrap it in a try/catch. The () brackets make this a generator expression rather than a list comprehension. Personally though I'd just use the regular filter/comprehension and take the first element (if there is one). These raise an exception if nothing is found filter(lambda x: x % 2 == 0, lst)[0] [x for x in lst if x %2 == 0][0] These return empty lists filter(lambda x: x % 2 == 0, lst)[:1] [x for x in lst if x %2 == 0][:1]


Instruction:
Unsupported operand type(s) for *: 'float' and 'Decimal'

I'm just playing around learning classes functions, etc., so I decided to create a simple function that should give me a tax amount. This is my code so far. class VAT_calculator: """ A set of methods for VAT calculations. """ def __init__(self, amount=None): self.amount = amount self.VAT = decimal.Decimal('0.095') def initialize(self): self.amount = 0 def total_with_VAT(self): """ Returns amount with VAT added. """ if not self.amount: msg = u"Cannot add VAT if no amount is passed!'" raise ValidationError(msg) return (self.amount * self.VAT).quantize(self.amount, rounding=decimal.ROUND_UP) My issue is I'm getting the following error: unsupported operand type(s) for : 'float' and 'Decimal'* Why should this not work?

Output: It seems like self.VAT is of decimal.Decimal type and self.amount is a float, a thing that you can't do. Try decimal.Decimal(self.amount) * self.VAT instead.


Instruction:
Trouble passing in lambda to apply for pandas DataFrame: "TypeError: <lambda>() got an unexpected keyword argument 'axis' "

I'm trying to apply a function to all rows of a pandas DataFrame (actually just one column in that DataFrame) I'm sure this is a syntax error but I'm know sure what I'm doing wrong df['col'].apply(lambda x, y:(x - y).total_seconds(), args=[d1], axis=1) The col column contains a bunch a datetime.datetime objects and d1 is the earliest of them. I'm trying to get a column of the total number of seconds for each of the rows. I keep getting the following error TypeError: <lambda>() got an unexpected keyword argument 'axis' I don't understand why axis is getting passed to my lambda function I've also tried doing def diff_dates(d1, d2): return (d1-d2).total_seconds() df['col'].apply(diff_dates, args=[d1], axis=1) And I get the same error.

Output: Note there is no axis param for a Series.apply call, as distinct to a DataFrame.apply call. Series.apply(func, convert_dtype=True, args=(), **kwds) ... func : function convert_dtype : boolean, default True Try to find better dtype for elementwise function results. If False, leave as dtype=object args : tuple Positional arguments to pass to function in addition to the value **kwds Additional keyword arguments passed to func. There is one for a df but it's unclear how you're expecting this to work when you're calling it on a series but you're expecting it to work on a row?


Instruction:
for x in y(): how does this work?

I was looking for code to spin a cursor in the terminal and found this. I was wondering what was happening in the code. In particular for c in spinning_cursor(): I've never seen this syntax. Is it because I am returning one element from a generator at a time with yield, and this is assigned to c? Any other examples of this for x in y() use? import sys import time def spinning_cursor(): cursor='/-\|' i = 0 while 1: yield cursor[i] i = (i + 1) % len(cursor) for c in spinning_cursor(): sys.stdout.write(c) sys.stdout.flush() time.sleep(0.1) sys.stdout.write('\b')

Output: Using yield turns a function into a generator. A generator is a specialized type of iterator. for always loops over iterables, taking each element in turn and assigning it to the name(s) you listed. spinning_cursor() returns a generator, the code inside spinning_cursor() doesn't actually run until you start iterating over the generator. Iterating over a generator means the code in the function is executed until it comes across a yield statement, at which point the result of the expression there is returned as the next value and execution is paused again. The for loop does just that, it'll call the equivalent of next() on the generator, until the generator signals it is done by raising StopIteration (which happens when the function returns). Each return value of next() is assigned, in turn, to c. You can see this by creating the generator on in the Python prompt: >>> def spinning_cursor(): ... cursor='/-\|' ... i = 0 ... while 1: ... yield cursor[i] ... i = (i + 1) % len(cursor) ... >>> sc = spinning_cursor() >>> sc <generator object spinning_cursor at 0x107a55eb0> >>> next(sc) '/' >>> next(sc) '-' >>> next(sc) '\\' >>> next(sc) '|' This specific generator never returns, so StopIteration is never raised and the for loop will go on forever unless you kill the script. A far more boring (but more efficient) alternative would be to use itertools.cycle(): from itertools import cycle spinning_cursor = cycle('/-\|')


Instruction:
Comma separated lists in django templates

If fruits is the list ['apples', 'oranges', 'pears'], is there a quick way using django template tags to produce "apples, oranges, and pears"? I know it's not difficult to do this using a loop and {% if counter.last %} statements, but because I'm going to use this repeatedly I think I'm going to have to learn how to write custom tags filters, and I don't want to reinvent the wheel if it's already been done. As an extension, my attempts to drop the Oxford Comma (ie return "apples, oranges and pears") are even messier.

Output: Here's the filter I wrote to solve my problem (it doesn't include the Oxford comma) def join_with_commas(obj_list): """Takes a list of objects and returns their string representations, separated by commas and with 'and' between the penultimate and final items For example, for a list of fruit objects: [<Fruit: apples>, <Fruit: oranges>, <Fruit: pears>] -> 'apples, oranges and pears' """ if not obj_list: return "" l=len(obj_list) if l==1: return u"%s" % obj_list[0] else: return ", ".join(str(obj) for obj in obj_list[:l-1]) \ + " and " + str(obj_list[l-1]) To use it in the template: {{ fruits|join_with_commas }}


Instruction:
Setting options from environment variables when using argparse

I have a script which has certain options that can either be passed on the command line, or from environment variables. The CLI should take precedence if both are present, and an error occur if neither are set. I could check that the option is assigned after parsing, but I prefer to let argparse to do the heavy lifting and be responsible for displaying the usage statement if parsing fails. I have come up with a couple of alternative approaches to this (which I will post below as answers so they can be discussed separately) but they feel pretty kludgey to me and I think that I am missing something. Is there an accepted "best" way of doing this? (Edit to make the desired behaviour clear when both the CLI option and environment variable are unset)

Output: I use this pattern frequently enough that I have packaged a simple action class to handle it: import argparse import os class EnvDefault(argparse.Action): def __init__(self, envvar, required=True, default=None, **kwargs): if envvar: if envvar in os.environ: default = os.environ[envvar] if required and default: required = False super(EnvDefault, self).__init__(default=default, required=required, **kwargs) def __call__(self, parser, namespace, values, option_string=None): setattr(namespace, self.dest, values) I can then call this from my code with: import argparse from envdefault import EnvDefault parser=argparse.ArgumentParser() parser.add_argument( "-u", "--url", action=EnvDefault, envvar='URL', help="Specify the URL to process (can also be specified using URL environment variable)") args=parser.parse_args()


Instruction:
Calculate cosine similarity given 2 sentence strings

From Python: tf-idf-cosine: to find document similarity , it is possible to calculate document similarity using tf-idf cosine. Without importing external libraries, are that any ways to calculate cosine similarity between 2 strings? s1 = "This is a foo bar sentence ." s2 = "This sentence is similar to a foo bar sentence ." s3 = "What is this string ? Totally not related to the other two lines ." cosine_sim(s1, s2) # Should give high cosine similarity cosine_sim(s1, s3) # Shouldn't give high cosine similarity value cosine_sim(s2, s3) # Shouldn't give high cosine similarity value

Output: A simple pure-Python implementation would be: import math import re from collections import Counter WORD = re.compile(r"\w+") def get_cosine(vec1, vec2): intersection = set(vec1.keys()) & set(vec2.keys()) numerator = sum([vec1[x] * vec2[x] for x in intersection]) sum1 = sum([vec1[x] ** 2 for x in list(vec1.keys())]) sum2 = sum([vec2[x] ** 2 for x in list(vec2.keys())]) denominator = math.sqrt(sum1) * math.sqrt(sum2) if not denominator: return 0.0 else: return float(numerator) / denominator def text_to_vector(text): words = WORD.findall(text) return Counter(words) text1 = "This is a foo bar sentence ." text2 = "This sentence is similar to a foo bar sentence ." vector1 = text_to_vector(text1) vector2 = text_to_vector(text2) cosine = get_cosine(vector1, vector2) print("Cosine:", cosine) Prints: Cosine: 0.861640436855 The cosine formula used here is described here. This does not include weighting of the words by tf-idf, but in order to use tf-idf, you need to have a reasonably large corpus from which to estimate tfidf weights. You can also develop it further, by using a more sophisticated way to extract words from a piece of text, stem or lemmatise it, etc.


Instruction:
pandas read_csv index_col=None not working with delimiters at the end of each line

I am going through the 'Python for Data Analysis' book and having trouble in the 'Example: 2012 Federal Election Commision Database' section reading the data to a DataFrame. The trouble is that one of the columns of data is always being set as the index column, even when the index_col argument is set to None. Here is the link to the data : http://www.fec.gov/disclosurep/PDownload.do. Here is the loading code (to save time in the checking, I set the nrows=10): import pandas as pd fec = pd.read_csv('P00000001-ALL.csv',nrows=10,index_col=None) To keep it short I am excluding the data column outputs, but here is my output (please not the Index values): In [20]: fec Out[20]: <class 'pandas.core.frame.DataFrame'> Index: 10 entries, C00410118 to C00410118 Data columns: ... dtypes: float64(4), int64(3), object(11) And here is the book's output (again with data columns excluded): In [13]: fec = read_csv('P00000001-ALL.csv') In [14]: fec Out[14]: <class 'pandas.core.frame.DataFrame'> Int64Index: 1001731 entries, 0 to 1001730 ... dtypes: float64(1), int64(1), object(14) The Index values in my output are actually the first column of data in the file, which is then moving all the rest of the data to the left by one. Would anyone know how to prevent this column of data to be listed as an index? I would like to have the index just +1 increasing integers. I am fairly new to python and pandas, so I apologize for any inconvenience. Thanks.

Output: Quick Answer Use index_col=False instead of index_col=None when you have delimiters at the end of each line to turn off index column inference and discard the last column. More Detail After looking at the data, there is a comma at the end of each line. And this quote (the documentation has been edited since the time this post was created): index_col: column number, column name, or list of column numbers/names, to use as the index (row labels) of the resulting DataFrame. By default, it will number the rows without using any column, unless there is one more data column than there are headers, in which case the first column is taken as the index. from the documentation shows that pandas believes you have n headers and n+1 data columns and is treating the first column as the index. EDIT 10/20/2014 - More information I found another valuable entry that is specifically about trailing limiters and how to simply ignore them: If a file has one more column of data than the number of column names, the first column will be used as the DataFrame’s row names: ... Ordinarily, you can achieve this behavior using the index_col option. There are some exception cases when a file has been prepared with delimiters at the end of each data line, confusing the parser. To explicitly disable the index column inference and discard the last column, pass index_col=False: ...


Instruction:
Non-blocking console input?

I am trying to make a simple IRC client in Python (as kind of a project while I learn the language). I have a loop that I use to receive and parse what the IRC server sends me, but if I use raw_input to input stuff, it stops the loop dead in its tracks until I input something (obviously). How can I input something without the loop stopping? (I don't think I need to post the code, I just want to input something without the while 1: loop stopping.) I'm on Windows.

Output: For Windows, console only, use the msvcrt module: import msvcrt num = 0 done = False while not done: print(num) num += 1 if msvcrt.kbhit(): print "you pressed",msvcrt.getch(),"so now i will quit" done = True For Linux, this article describes the following solution, it requires the termios module: import sys import select import tty import termios def isData(): return select.select([sys.stdin], [], [], 0) == ([sys.stdin], [], []) old_settings = termios.tcgetattr(sys.stdin) try: tty.setcbreak(sys.stdin.fileno()) i = 0 while 1: print(i) i += 1 if isData(): c = sys.stdin.read(1) if c == '\x1b': # x1b is ESC break finally: termios.tcsetattr(sys.stdin, termios.TCSADRAIN, old_settings) For cross platform, or in case you want a GUI as well, you can use Pygame: import pygame from pygame.locals import * def display(str): text = font.render(str, True, (255, 255, 255), (159, 182, 205)) textRect = text.get_rect() textRect.centerx = screen.get_rect().centerx textRect.centery = screen.get_rect().centery screen.blit(text, textRect) pygame.display.update() pygame.init() screen = pygame.display.set_mode( (640,480) ) pygame.display.set_caption('Python numbers') screen.fill((159, 182, 205)) font = pygame.font.Font(None, 17) num = 0 done = False while not done: display( str(num) ) num += 1 pygame.event.pump() keys = pygame.key.get_pressed() if keys[K_ESCAPE]: done = True


Instruction:
How to take partial screenshot with Selenium WebDriver in python?

I have searched a lot for this but couldn't find a solution. Here's a similar question with a possible solution in java. Is there a similar solution in Python?

Output: Other than Selenium, this example also requires the PIL Imaging library. Sometimes this is put in as one of the standard libraries and sometimes it's not, but if you don't have it you can install it with pip install Pillow from selenium import webdriver from PIL import Image from io import BytesIO fox = webdriver.Firefox() fox.get('http://stackoverflow.com/') # now that we have the preliminary stuff out of the way time to get that image :D element = fox.find_element_by_id('hlogo') # find part of the page you want image of location = element.location size = element.size png = fox.get_screenshot_as_png() # saves screenshot of entire page fox.quit() im = Image.open(BytesIO(png)) # uses PIL library to open image in memory left = location['x'] top = location['y'] right = location['x'] + size['width'] bottom = location['y'] + size['height'] im = im.crop((left, top, right, bottom)) # defines crop points im.save('screenshot.png') # saves new cropped image and finally the output is... the Stackoverflow logo!!! Now of course this would be overkill for just grabbing a static image but if your want to grab something that requires Javascript to get to this could be a viable solution.


Instruction:
How to split strings into text and number?

I'd like to split strings like these 'foofo21' 'bar432' 'foobar12345' into ['foofo', '21'] ['bar', '432'] ['foobar', '12345'] Does somebody know an easy and simple way to do this in python?

Output: I would approach this by using re.match in the following way: import re match = re.match(r"([a-z]+)([0-9]+)", 'foofo21', re.I) if match: items = match.groups() print(items) >> ("foofo", "21")


Instruction:
How to see sizes of installed pip packages?

On Linux Debian, how can I list all installed python pip packages and the size (amount of disk space used) that each one takes up?

Output: Go to the package site to find the size e.g. https://pypi.python.org/pypi/pip/json Then expand releases, find the version, and look up the size (in bytes).


Instruction:
Built-in module to calculate the least common multiple

I am currently using a function that accepts two numbers and uses a loop to find the least common multiple of those numbers, def lcm(x, y): """This function takes two integers and returns the L.C.M.""" # Choose the greater number if x > y: greater = x else: greater = y while(True): if((greater % x == 0) and (greater % y == 0)): lcm = greater break greater += 1 return lcm Is there a built-in module in Python that does it instead of writing a custom function?

Output: In Python 3.8 and earlier There is no such thing built into the stdlib. However, there is a Greatest Common Divisor function in the math library. (For Python 3.4 or 2.7, it's buried in fractions instead.) And writing an LCM on top of a GCD is pretty trivial: def lcm(a, b): return abs(a*b) // math.gcd(a, b) Or, if you're using NumPy, it's come with an lcm function for quite some time now.


Instruction:
Flipping the boolean values in a list Python

I have a boolean list in Python mylist = [True , True, False,...] which I want to change to the logical opposite [False, False, True , ...] Is there an inbuilt way to do this in Python (something like a call not(mylist) ) without a hand-written loop to reverse the elements?

Output: It's easy with list comprehension: mylist = [True , True, False] [not elem for elem in mylist] yields [False, False, True]


Instruction:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 35: invalid start byte

I am trying to read a CSV file using the below script. Past = pd.read_csv("C:/Users/Admin/Desktop/Python/Past.csv", encoding='utf-8') But, I get the error "UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 35: invalid start byte" Where is the issue? I used encoding in the script and thought it would resolve the error.

Output: This happens because you chose the wrong encoding. Since you are working on a Windows machine, just replacing Past = pd.read_csv("C:/Users/.../Past.csv", encoding='utf-8') with Past = pd.read_csv("C:/Users/.../Past.csv", encoding='cp1252') should solve the problem.