Instruction:
"Fire and forget" python async/await

Sometimes there is some non-critical asynchronous operation that needs to happen but I don't want to wait for it to complete. In Tornado's coroutine implementation you can "fire & forget" an asynchronous function by simply ommitting the yield key-word. I've been trying to figure out how to "fire & forget" with the new async/await syntax released in Python 3.5. E.g., a simplified code snippet: async def async_foo(): print("Do some stuff asynchronously here...") def bar(): async_foo() # fire and forget "async_foo()" bar() What happens though is that bar() never executes and instead we get a runtime warning: RuntimeWarning: coroutine 'async_foo' was never awaited async_foo() # fire and forget "async_foo()"

Output: Editor's note: According to a comment below, According to the documentation the event loop only keeps a "weak reference" to a task, see https://docs.python.org/3.13/library/asyncio-task.html#creating-tasks so it appears to me that you MUST save a reference in order to ensure that the task is not destroyed before it is done. It seems to me that this makes this answer wrong. Upd: Replace asyncio.ensure_future with asyncio.create_task everywhere if you're using Python >= 3.7 It's a newer, nicer way to spawn tasks. asyncio.Task to "fire and forget" According to python docs for asyncio.Task it is possible to start some coroutine to execute "in the background". The task created by asyncio.ensure_future won't block the execution (therefore the function will return immediately!). This looks like a way to "fire and forget" as you requested. import asyncio async def async_foo(): print("async_foo started") await asyncio.sleep(1) print("async_foo done") async def main(): asyncio.ensure_future(async_foo()) # fire and forget async_foo() # btw, you can also create tasks inside non-async funcs print('Do some actions 1') await asyncio.sleep(1) print('Do some actions 2') await asyncio.sleep(1) print('Do some actions 3') if __name__ == '__main__': loop = asyncio.get_event_loop() loop.run_until_complete(main()) Output: Do some actions 1 async_foo started Do some actions 2 async_foo done Do some actions 3 What if tasks are executing after the event loop has completed? Note that asyncio expects tasks to be completed at the moment the event loop completes. So if you'll change main() to: async def main(): asyncio.ensure_future(async_foo()) # fire and forget print('Do some actions 1') await asyncio.sleep(0.1) print('Do some actions 2') You'll get this warning after the program finished: Task was destroyed but it is pending! task: <Task pending coro=<async_foo() running at [...] To prevent that you can just await all pending tasks after the event loop has completed: async def main(): asyncio.ensure_future(async_foo()) # fire and forget print('Do some actions 1') await asyncio.sleep(0.1) print('Do some actions 2') if __name__ == '__main__': loop = asyncio.get_event_loop() loop.run_until_complete(main()) # Let's also finish all running tasks: pending = asyncio.Task.all_tasks() loop.run_until_complete(asyncio.gather(*pending)) Kill tasks instead of awaiting them Sometimes you don't want to await tasks to be done (for example, some tasks may be created to run forever). In that case, you can just cancel() them instead of awaiting them: import asyncio from contextlib import suppress async def echo_forever(): while True: print("echo") await asyncio.sleep(1) async def main(): asyncio.ensure_future(echo_forever()) # fire and forget print('Do some actions 1') await asyncio.sleep(1) print('Do some actions 2') await asyncio.sleep(1) print('Do some actions 3') if __name__ == '__main__': loop = asyncio.get_event_loop() loop.run_until_complete(main()) # Let's also cancel all running tasks: pending = asyncio.Task.all_tasks() for task in pending: task.cancel() # Now we should await task to execute it's cancellation. # Cancelled task raises asyncio.CancelledError that we can suppress: with suppress(asyncio.CancelledError): loop.run_until_complete(task) Output: Do some actions 1 echo Do some actions 2 echo Do some actions 3 echo


Instruction:
Combine Date and Time columns using pandas

I have a pandas dataframe with the following columns: data = {'Date': ['01-06-2013', '02-06-2013', '02-06-2013', '02-06-2013', '02-06-2013', '03-06-2013', '03-06-2013', '03-06-2013', '03-06-2013', '04-06-2013'], 'Time': ['23:00:00', '01:00:00', '21:00:00', '22:00:00', '23:00:00', '01:00:00', '21:00:00', '22:00:00', '23:00:00', '01:00:00']} df = pd.DataFrame(data) Date Time 0 01-06-2013 23:00:00 1 02-06-2013 01:00:00 2 02-06-2013 21:00:00 3 02-06-2013 22:00:00 4 02-06-2013 23:00:00 5 03-06-2013 01:00:00 6 03-06-2013 21:00:00 7 03-06-2013 22:00:00 8 03-06-2013 23:00:00 9 04-06-2013 01:00:00 How do I combine data['Date'] & data['Time'] to get the following? Is there a way of doing it using pd.to_datetime? Date 01-06-2013 23:00:00 02-06-2013 01:00:00 02-06-2013 21:00:00 02-06-2013 22:00:00 02-06-2013 23:00:00 03-06-2013 01:00:00 03-06-2013 21:00:00 03-06-2013 22:00:00 03-06-2013 23:00:00 04-06-2013 01:00:00

Output: It's worth mentioning that you may have been able to read this in directly e.g. if you were using read_csv using parse_dates=[['Date', 'Time']]. Assuming these are just strings you could simply add them together (with a space), allowing you to use to_datetime, which works without specifying the format= parameter In [11]: df['Date'] + ' ' + df['Time'] Out[11]: 0 01-06-2013 23:00:00 1 02-06-2013 01:00:00 2 02-06-2013 21:00:00 3 02-06-2013 22:00:00 4 02-06-2013 23:00:00 5 03-06-2013 01:00:00 6 03-06-2013 21:00:00 7 03-06-2013 22:00:00 8 03-06-2013 23:00:00 9 04-06-2013 01:00:00 dtype: object In [12]: pd.to_datetime(df['Date'] + ' ' + df['Time']) Out[12]: 0 2013-01-06 23:00:00 1 2013-02-06 01:00:00 2 2013-02-06 21:00:00 3 2013-02-06 22:00:00 4 2013-02-06 23:00:00 5 2013-03-06 01:00:00 6 2013-03-06 21:00:00 7 2013-03-06 22:00:00 8 2013-03-06 23:00:00 9 2013-04-06 01:00:00 dtype: datetime64[ns] Alternatively, without the + ' ', but the format= parameter must be used. Additionally, pandas is good at inferring the format to be converted to a datetime, however, specifying the exact format is faster. pd.to_datetime(df['Date'] + df['Time'], format='%m-%d-%Y%H:%M:%S') Note: surprisingly (for me), this works fine with NaNs being converted to NaT, but it is worth worrying that the conversion (perhaps using the raise argument). %%timeit # sample dataframe with 10000000 rows using df from the OP df = pd.concat([df for _ in range(1000000)]).reset_index(drop=True) %%timeit pd.to_datetime(df['Date'] + ' ' + df['Time']) [result]: 1.73 s ± 10.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) %%timeit pd.to_datetime(df['Date'] + df['Time'], format='%m-%d-%Y%H:%M:%S') [result]: 1.33 s ± 9.88 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)


Instruction:
How to expand a list to function arguments in Python

Is there syntax that allows you to expand a list into the arguments of a function call? Example: # Trivial example function, not meant to do anything useful. def foo(x,y,z): return "%d, %d, %d" %(x,y,z) # List of values that I want to pass into foo. values = [1,2,3] # I want to do something like this, and get the result "1, 2, 3": foo( values.howDoYouExpandMe() )

Output: It exists, but it's hard to search for. I think most people call it the "splat" operator. It's in the documentation as "Unpacking argument lists". You'd use it like this for positional arguments: values = [1, 2] foo(*values) There's also one for dictionaries to call with named arguments: d = {'a': 1, 'b': 2} def foo(a, b): pass foo(**d)


Instruction:
Wheel file installation

How do I install a .whl file? I have the wheel library, but I don't know how to use it to install those files. I have the .whl file, but I don't know how to run it.

Output: You normally use a tool like pip to install wheels. Leave it to the tool to discover and download the file if this is for a project hosted on PyPI. For this to work, you do need to install the wheel package: pip install wheel You can then tell pip to install the project (and it'll download the wheel if available), or the wheel file directly: pip install project_name # discover, download and install pip install wheel_file.whl # directly install the wheel Also see the wheel project documentation.


Instruction:
How to convert list of numpy arrays into single numpy array?

Suppose I have ; LIST = [[array([1, 2, 3, 4, 5]), array([1, 2, 3, 4, 5],[1,2,3,4,5])] # inner lists are numpy arrays I try to convert; array([[1, 2, 3, 4, 5], [1, 2, 3, 4, 5], [1, 2, 3, 4, 5]) I am solving it by iteration on vstack right now but it is really slow for especially large LIST What do you suggest for the best efficient way?

Output: In general you can concatenate a whole sequence of arrays along any axis: numpy.concatenate( LIST, axis=0 ) but you do have to worry about the shape and dimensionality of each array in the list (for a 2-dimensional 3x5 output, you need to ensure that they are all 2-dimensional n-by-5 arrays already). If you want to concatenate 1-dimensional arrays as the rows of a 2-dimensional output, you need to expand their dimensionality. As Jorge's answer points out, there is also the function stack, introduced in numpy 1.10: numpy.stack( LIST, axis=0 ) This takes the complementary approach: it creates a new view of each input array and adds an extra dimension (in this case, on the left, so each n-element 1D array becomes a 1-by-n 2D array) before concatenating. It will only work if all the input arrays have the same shape. vstack (or equivalently row_stack) is often an easier-to-use solution because it will take a sequence of 1- and/or 2-dimensional arrays and expand the dimensionality automatically where necessary and only where necessary, before concatenating the whole list together. Where a new dimension is required, it is added on the left. Again, you can concatenate a whole list at once without needing to iterate: numpy.vstack( LIST ) This flexible behavior is also exhibited by the syntactic shortcut numpy.r_[ array1, ...., arrayN ] (note the square brackets). This is good for concatenating a few explicitly-named arrays but it becomes less readable in your situation because [] subscripting will not accept a list. You would need to convert your sequence to a tuple: numpy.r_[tuple(LIST)]. It's more readable to simply use vstack(). There is also an analogous function column_stack and shortcut c_[...], for horizontal (column-wise) stacking, as well as an almost-analogous function hstack—although for some reason the latter is less flexible (it is stricter about input arrays' dimensionality, and tries to concatenate 1-D arrays end-to-end instead of treating them as columns). Finally, in the specific case of vertical stacking of 1-D arrays, the following also works: numpy.array( LIST ) ...because arrays can be constructed out of a sequence of other arrays, adding a new dimension to the beginning.


Instruction:
In python, why use logging instead of print?

For simple debugging in a complex project is there a reason to use the python logger instead of print? What about other use-cases? Is there an accepted best use-case for each (especially when you're only looking for stdout)? I've always heard that this is a "best practice" but I haven't been able to figure out why.

Output: The logging package has a lot of useful features: Easy to see where and when (even what line no.) a logging call is being made from. You can log to files, sockets, pretty much anything, all at the same time. You can differentiate your logging based on severity. Print doesn't have any of these. Also, if your project is meant to be imported by other python tools, it's bad practice for your package to print things to stdout, since the user likely won't know where the print messages are coming from. With logging, users of your package can choose whether they want to propogate logging messages from your tool or not.


Instruction:
Can't import my own modules in Python

I'm having a hard time understanding how module importing works in Python (I've never done it in any other language before either). Let's say I have: myapp/__init__.py myapp/myapp/myapp.py myapp/myapp/SomeObject.py myapp/tests/TestCase.py Now I'm trying to get something like this: myapp.py =================== from myapp import SomeObject # stuff ... TestCase.py =================== from myapp import SomeObject # some tests on SomeObject However, I'm definitely doing something wrong as Python can't see that myapp is a module: ImportError: No module named myapp

Output: In your particular case it looks like you're trying to import SomeObject from the myapp.py and TestCase.py scripts. From myapp.py, do import SomeObject since it is in the same folder. For TestCase.py, do from ..myapp import SomeObject However, this will work only if you are importing TestCase from the package. If you want to directly run python TestCase.py, you would have to mess with your path. This can be done within Python: import sys sys.path.append("..") from myapp import SomeObject though that is generally not recommended. In general, if you want other people to use your Python package, you should use distutils to create a setup script. That way, anyone can install your package easily using a command like python setup.py install and it will be available everywhere on their machine. If you're serious about the package, you could even add it to the Python Package Index, PyPI.


Instruction:
How to properly use unit-testing's assertRaises() with NoneType objects

I did a simple test case: def setUp(self): self.testListNone = None def testListSlicing(self): self.assertRaises(TypeError, self.testListNone[:1]) And I am expecting the test to pass, but I am getting an exception: Traceback (most recent call last): self.assertRaises(TypeError, self.testListNone[:1]) TypeError: 'NoneType' object is unsubscriptable I thought that assertRaises will pass since TypeError exception will be raised. What is the explanation?

Output: If you are using Python 2.7 or above, you can use the ability of assertRaises to be used as a context manager and do: with self.assertRaises(TypeError): self.testListNone[:1] If you are using Python 2.6, another way beside the one given until now is to use unittest2 which is a back port of unittest's new features to Python 2.6, and you can make it work using the code above. N.B: I'm a big fan of the new feature (SkipTest, test discovery, etc.) of unittest, so I intend to use unittest2 as much as I can. I advise to do the same, because there is a lot more than what unittest come with in Python 2.6 or lower.


Instruction:
Python - Extract a PDF page as a jpeg

How can I efficiently save a particular page of a PDF as a jpeg file using Python? I have a Python Flask web server where PDFs will be uploaded and I want to also store jpeg files that correspond to each PDF page. This solution is close but it does not result in the entire page being converted to a jpeg.

Output: The pdf2image library can be used. You can install it simply using, pip install pdf2image Once installed you can use following code to get images. from pdf2image import convert_from_path pages = convert_from_path('pdf_file', 500) Saving pages in jpeg format for count, page in enumerate(pages): page.save(f'out{count}.jpg', 'JPEG') Edit: the Github repo pdf2image also mentions that it uses pdftoppm and that it requires other installations: pdftoppm is the piece of software that does the actual magic. It is distributed as part of a greater package called poppler. Windows users will have to install [poppler for Windows] see ** below Mac users will have to install poppler for Mac. Linux users will have pdftoppm pre-installed with the distro (Tested on Ubuntu and Archlinux) if it's not, run sudo apt install poppler-utils. You can install the latest version under Windows using anaconda by doing: conda install -c conda-forge poppler ** note: Windows 64 bit versions upto 24.08 are available at https://github.com/oschwartz10612/poppler-windows but note that for 32 bit 22.02 was the last one included in TeXLive 2022 (https://poppler.freedesktop.org/releases.html) so you'll not be getting the latest features or bug fixes.


Instruction:
Iterate over model instance field names and values in template

I'm trying to create a basic template to display the selected instance's field values, along with their names. Think of it as just a standard output of the values of that instance in table format, with the field name (verbose_name specifically if specified on the field) in the first column and the value of that field in the second column. For example, let's say we have the following model definition: class Client(Model): name = CharField(max_length=150) email = EmailField(max_length=100, verbose_name="E-mail") I would want it to be output in the template like so (assume an instance with the given values): Field Name Field Value ---------- ----------- Name Wayne Koorts E-mail waynes@email.com What I'm trying to achieve is being able to pass an instance of the model to a template and be able to iterate over it dynamically in the template, something like this:
 {% for field in fields %}
 {{ field.name }}	 {{ field.value }}
 {% endfor %}
 Is there a neat, "Django-approved" way to do this? It seems like a very common task, and I will need to do it often for this particular project.

Output: Since Django 1.9: use model._meta.get_fields() to get the model's fields and field.name to get each field name. Previous to Django 1.9: model._meta.get_all_field_names() will give you all the model's field names, then you can use model._meta.get_field() to work your way to the verbose name, and getattr(model_instance, 'field_name') to get the value from the model.


Instruction:
Handling very large numbers in Python

I've been considering fast poker hand evaluation in Python. It occurred to me that one way to speed the process up would be to represent all the card faces and suits as prime numbers and multiply them together to represent the hands. To whit: class PokerCard: faces = '23456789TJQKA' suits = 'cdhs' facePrimes = [11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 53, 59, 61] suitPrimes = [2, 3, 5, 7] AND def HashVal(self): return PokerCard.facePrimes[self.cardFace] * PokerCard.suitPrimes[self.cardSuit] This would give each hand a numeric value that, through modulo could tell me how many kings are in the hand or how many hearts. For example, any hand with five or more clubs in it would divide evenly by 2^5; any hand with four kings would divide evenly by 59^4, etc. The problem is that a seven-card hand like AcAdAhAsKdKhKs has a hash value of approximately 62.7 quadrillion, which would take considerably more than 32 bits to represent internally. Is there a way to store such large numbers in Python that will allow me to perform arithmetic operations on it?

Output: Python supports a "bignum" integer type which can work with arbitrarily large numbers. In Python 2.5+, this type is called long and is separate from the int type, but the interpreter will automatically use whichever is more appropriate. In Python 3.0+, long has been renamed int and the old int type has been dropped completely. That's just an implementation detail, though — as long as you have version 2.5 or better, just perform standard math operations and any number which exceeds the boundaries of 32-bit math will be automatically (and transparently) converted to a bignum. You can find all the gory details in PEP 0237.


Instruction:
What is the most efficient string concatenation method in Python?

Is there an efficient mass string concatenation method in Python (like StringBuilder in C# or StringBuffer in Java)? I found following methods here: Simple concatenation using + Using a string list and the join method Using UserString from the MutableString module Using a character array and the array module Using cStringIO from the StringIO module What should be used and why? (A related question is here.)

Output: If you know all components beforehand once, use the literal string interpolation, also known as f-strings or formatted strings, introduced in Python 3.6. Given the test case from mkoistinen's answer, having strings domain = 'some_really_long_example.com' lang = 'en' path = 'some/really/long/path/' The contenders and their execution time on my computer using Python 3.6 on Linux as timed by IPython and the timeit module are f'http://{domain}/{lang}/{path}' - 0.151 µs 'http://%s/%s/%s' % (domain, lang, path) - 0.321 µs 'http://' + domain + '/' + lang + '/' + path - 0.356 µs ''.join(('http://', domain, '/', lang, '/', path)) - 0.249 µs (notice that building a constant-length tuple is slightly faster than building a constant-length list). Thus the shortest and the most beautiful code possible is also fastest. The speed can be contrasted with the fastest method for Python 2, which is + concatenation on my computer; and that takes 0.203 µs with 8-bit strings, and 0.259 µs if the strings are all Unicode. (In alpha versions of Python 3.6 the implementation of f'' strings was the slowest possible - actually the generated byte code is pretty much equivalent to the ''.join() case with unnecessary calls to str.__format__ which without arguments would just return self unchanged. These inefficiencies were addressed before 3.6 final.)


Instruction:
Pythonic way to check if a list is sorted or not

Is there a pythonic way to check if a list is already sorted in ASC or DESC listtimestamps = [1, 2, 3, 5, 6, 7] something like isttimestamps.isSorted() that returns True or False. I want to input a list of timestamps for some messages and check if the the transactions appeared in the correct order.

Output: Here is a one liner: all(l[i] <= l[i+1] for i in range(len(l) - 1)) If using Python 2, use xrange instead of range. For reverse=True, use >= instead of <=.


Instruction:
How to display an image

I tried to use IPython.display with the following code: from IPython.display import display, Image display(Image(filename='MyImage.png')) I also tried to use matplotlib with the following code: import matplotlib.pyplot as plt import matplotlib.image as mpimg plt.imshow(mpimg.imread('MyImage.png')) In both cases, nothing is displayed, not even an error message.

Output: If you are using matplotlib and want to show the image in your interactive notebook, try the following: %matplotlib inline import matplotlib.pyplot as plt import matplotlib.image as mpimg img = mpimg.imread('your_image.png') imgplot = plt.imshow(img) plt.show()


Instruction:
Select 50 items from list at random

I have a function which reads a list of items from a file. How can I select only 50 items from the list randomly to write to another file? def randomizer(input, output='random.txt'): query = open(input).read().split() out_file = open(output, 'w') random.shuffle(query) for item in query: out_file.write(item + '\n') For example, if the total randomization file was random_total = ['9', '2', '3', '1', '5', '6', '8', '7', '0', '4'] and I would want a random set of 3, the result could be random = ['9', '2', '3'] How can I select 50 from the list that I randomized? Even better, how could I select 50 at random from the original list?

Output: If the list is in random order, you can just take the first 50. Otherwise, use import random random.sample(the_list, 50) random.sample help text: sample(self, population, k) method of random.Random instance Chooses k unique random elements from a population sequence. Returns a new list containing elements from the population while leaving the original population unchanged. The resulting list is in selection order so that all sub-slices will also be valid random samples. This allows raffle winners (the sample) to be partitioned into grand prize and second place winners (the subslices). Members of the population need not be hashable or unique. If the population contains repeats, then each occurrence is a possible selection in the sample. To choose a sample in a range of integers, use xrange as an argument. This is especially fast and space efficient for sampling from a large population: sample(xrange(10000000), 60)


Instruction:
What is the fastest way to check if a class has a function defined?

I'm writing an AI state space search algorithm, and I have a generic class which can be used to quickly implement a search algorithm. A subclass would define the necessary operations, and the algorithm does the rest. Here is where I get stuck: I want to avoid regenerating the parent state over and over again, so I have the following function, which returns the operations that can be legally applied to any state: def get_operations(self, include_parent=True): ops = self._get_operations() if not include_parent and self.path.parent_op: try: parent_inverse = self.invert_op(self.path.parent_op) ops.remove(parent_inverse) except NotImplementedError: pass return ops And the invert_op function throws by default. Is there a faster way to check to see if the function is not defined than catching an exception? I was thinking something on the lines of checking for present in dir, but that doesn't seem right. hasattr is implemented by calling getattr and checking if it raises, which is not what I want.

Output: Yes, use getattr() to get the attribute, and callable() to verify it is a method: invert_op = getattr(self, "invert_op", None) if callable(invert_op): invert_op(self.path.parent_op) Note that getattr() normally throws exception when the attribute doesn't exist. However, if you specify a default value (None, in this case), it will return that instead.


Instruction:
Binning a column with pandas

I have a data frame column with numeric values: df['percentage'].head() 46.5 44.2 100.0 42.12 I want to see the column as bin counts: bins = [0, 1, 5, 10, 25, 50, 100] How can I get the result as bins with their value counts? [0, 1] bin amount [1, 5] etc [5, 10] etc ...

Output: You can use pandas.cut: bins = [0, 1, 5, 10, 25, 50, 100] df['binned'] = pd.cut(df['percentage'], bins) print (df) percentage binned 0 46.50 (25, 50] 1 44.20 (25, 50] 2 100.00 (50, 100] 3 42.12 (25, 50] bins = [0, 1, 5, 10, 25, 50, 100] labels = [1,2,3,4,5,6] df['binned'] = pd.cut(df['percentage'], bins=bins, labels=labels) print (df) percentage binned 0 46.50 5 1 44.20 5 2 100.00 6 3 42.12 5 Or numpy.searchsorted: bins = [0, 1, 5, 10, 25, 50, 100] df['binned'] = np.searchsorted(bins, df['percentage'].values) print (df) percentage binned 0 46.50 5 1 44.20 5 2 100.00 6 3 42.12 5 ...and then value_counts or groupby and aggregate size: s = pd.cut(df['percentage'], bins=bins).value_counts() print (s) (25, 50] 3 (50, 100] 1 (10, 25] 0 (5, 10] 0 (1, 5] 0 (0, 1] 0 Name: percentage, dtype: int64 s = df.groupby(pd.cut(df['percentage'], bins=bins)).size() print (s) percentage (0, 1] 0 (1, 5] 0 (5, 10] 0 (10, 25] 0 (25, 50] 3 (50, 100] 1 dtype: int64 By default cut returns categorical. Series methods like Series.value_counts() will use all categories, even if some categories are not present in the data, operations in categorical.


Instruction:
What is the purpose of the __repr__ method?

For example, def __repr__(self): return '<%s %s (%s:%s) %s>' % ( self.__class__.__name__, self.urlconf_name, self.app_name, self.namespace, self.regex.pattern) What is the significance/purpose of this method?

Output: __repr__ should return a printable representation of the object, most likely one of the ways possible to create this object. See also documentation for repr() (which calls __repr__). __repr__ is more for developers, while __str__ is for end users. A simple example (interactive): class Point: def __init__(self, x, y): self.x = x self.y = y def __repr__(self): cls = self.__class__.__name__ return f'{cls}(x={self.x!r}, y={self.y!r})' p = Point(1, 2) p Output: Point(x=1, y=2)


Instruction:
Why is semicolon allowed in this Python snippet?

Python does not warrant the use of semicolons to end statements. So why is this (below) allowed? import pdb; pdb.set_trace()

Output: Python does not require semicolons to terminate statements. Semicolons can be used to delimit statements if you wish to put multiple statements on the same line. Now, why is this allowed? It's a simple design decision. I don't think Python needs this semicolon thing, but somebody thought it would be nice to have and added it to the language.


Instruction:
VSCode doesn't show poetry virtualenvs in select interpreter option

I need help. VSCode will NEVER find poetry virtualenv interpreter no matter what I try. Installed poetry Python package manager using a standard $ curl method as explained in the official documentation. Started a project by $ poetry new finance-essentials_37-64, installed poetry environment with $ poetry install. So now I can see that I indeed have a virtual environment by: Jaepil@Jaepil-PC MINGW64 /e/VSCodeProjects/finance_essentials_37-64 $ poetry env list >> finance-essentials-37-64-SCQrHB_N-py3.7 (Activated) and this virtualenv is installed at: C:\Users\Jaepil\AppData\Local\pypoetry\Cache\virtualenvs, which has finance-essentials-37-64-SCQrHB_N-py3.7 directory. However, VSCode is unable to find this virtualenv in its 'select interpreter' command. I only see a bunch of Anaconda and Pipenv environments but not the poetry environment's interpreter that I've just made. I also added "python.venvPath": "~/.cache/pypoetry/virtualenvs", to my settings.json as suggested in here, but to no avail. Still doesn't work. I also tried an absolute path, by adding "python.venvPath": "C:\\Users\\Jaepil\\AppData\\Local\\pypoetry\\Cache\\virtualenvs", to the same settings, but it also doesn't work. VSCode settings reference states that it has python.poetryPath as a default but it doesn't seem to work either. Should I change the default value "poetry" in this case? python.poetryPath "poetry" Specifies the location of the Poetry dependency manager executable, if installed. The default value "poetry" assumes the executable is in the current path. The Python extension uses this setting to install packages when Poetry is available and there's a poetry.lock file in the workspace folder. I'm on Windows 10 pro 64bit & Has Python 3.7.6 installed on the system. PS C:\Users\Jaepil> python Python 3.7.6 (tags/v3.7.6:43364a7ae0, Dec 19 2019, 00:42:30) [MSC v.1916 64 bit (AMD64)] on win32

Output: You just need to type in your shell: poetry config virtualenvs.in-project true The virtualenv will be created inside the project path and vscode will recognize. Consider adding this to your .bashrc or .zshrc. If you already have created your project, you need to re-create the virtualenv to make it appear in the correct place: poetry env list # shows the name of the current environment poetry env remove <current environment> poetry install # will create a new environment using your updated configuration


Instruction:
'virtualenv' won't activate on Windows

Essentially I cannot seem to activate my virtualenv environment which I create. I'm doing this inside of Windows PowerShell through using scripts\activate but I get an error message: "cannot be loaded because the execution of scripts is disabled on this system". Could this be because I don't carry administrator privileges on my computer?

Output: According to Microsoft Tech Support it might be a problem with Execution Policy Settings. To fix it, you should try executing Set-ExecutionPolicy Unrestricted -Scope Process (as mentioned in the comment section by @wtsiamruk) in your PowerShell window. This would allow running virtualenv in the current PowerShell session. There is also another approach that is more unsafe, but recommended by MS Tech Support. This approach would be to use Set-ExecutionPolicy Unrestricted -Force (which do unleash powers to screw Your system up). However, before you use this unsafe way, be sure to check what your current ExecutionPolicy setting is by using get-ExecutionPolicy. Then, when you are done, you can revert back to this ExecutionPolicy by using Set-ExecutionPolicy %the value the get-ExecutionPolicy command gave you% -Force.


Instruction:
How to import an existing requirements.txt into a Poetry project?

I am trying out Poetry in an existing project. It used pyenv and virtual env originally so I have a requirements.txt file with the project's dependencies. I want to import the requirements.txt file using Poetry, so that I can load the dependencies for the first time. I've looked through poetry's documentation, but I haven't found a way to do this. Is there a way to do it? I know that I can add all packages manually, but I was hoping for a more automated process, because there are a lot of packages.

Output: poetry doesn't support this directly. But if you have a handmade list of required packages (at best without any version numbers), that only contain the main dependencies and not the dependencies of a dependency you could do this: $ cat requirements.txt | xargs poetry add


Instruction:
python list by value not by reference

Let's take an example a=['help', 'copyright', 'credits', 'license'] b=a b.append('XYZ') b ['help', 'copyright', 'credits', 'license', 'XYZ'] a ['help', 'copyright', 'credits', 'license', 'XYZ'] I wanted to append value in list 'b' but the value of list 'a' have also changed. I think I have little idea why its like this (python passes lists by reference). My question is "how can I pass it by value so that appending 'b' does't change values in 'a' ?"

Output: You cannot pass anything by value in Python. If you want to make a copy of a, you can do so explicitly, as described in the official Python FAQ: b = a[:]


Instruction:
Duplicate log output when using Python logging module

I am using python logger. The following is my code: import os import time import datetime import logging class Logger : def myLogger(self): logger = logging.getLogger('ProvisioningPython') logger.setLevel(logging.DEBUG) now = datetime.datetime.now() handler=logging.FileHandler('/root/credentials/Logs/ProvisioningPython'+ now.strftime("%Y-%m-%d") +'.log') formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s') handler.setFormatter(formatter) logger.addHandler(handler) return logger The problem I have is that I get multiple entries in the log file for each logger.info call. How can I solve this?

Output: Function logging.getLogger() returns the same instance for a given name. The problem is that every time you call myLogger(), it's adding another handler to the instance, which causes the duplicate logs. Perhaps something like this? import os import time import datetime import logging loggers = {} def myLogger(name): global loggers if loggers.get(name): return loggers.get(name) else: logger = logging.getLogger(name) logger.setLevel(logging.DEBUG) now = datetime.datetime.now() handler = logging.FileHandler( '/root/credentials/Logs/ProvisioningPython' + now.strftime("%Y-%m-%d") + '.log') formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s') handler.setFormatter(formatter) logger.addHandler(handler) loggers[name] = logger return logger


Instruction:
ImportError: cannot import name 'url' from 'django.conf.urls' after upgrading to Django 4.0

After upgrading to Django 4.0, I get the following error when running python manage.py runserver ... File "/path/to/myproject/myproject/urls.py", line 16, in <module> from django.conf.urls import url ImportError: cannot import name 'url' from 'django.conf.urls' (/path/to/my/venv/lib/python3.9/site-packages/django/conf/urls/__init__.py) My urls.py is as follows: from django.conf.urls from myapp.views import home urlpatterns = [ url(r'^$', home, name="home"), url(r'^myapp/', include('myapp.urls'), ]

Output: django.conf.urls.url() was deprecated in Django 3.0, and is removed in Django 4.0+. The easiest fix is to replace url() with re_path(). re_path uses regexes like url, so you only have to update the import and replace url with re_path. from django.urls import include, re_path from myapp.views import home urlpatterns = [ re_path(r'^$', home, name='home'), re_path(r'^myapp/', include('myapp.urls'), ] Alternatively, you could switch to using path. path() does not use regexes, so you'll have to update your URL patterns if you switch to path. from django.urls import include, path from myapp.views import home urlpatterns = [ path('', home, name='home'), path('myapp/', include('myapp.urls'), ] If you have a large project with many URL patterns to update, you may find the django-upgrade library useful to update your urls.py files.


Instruction:
What is the difference between json.dump() and json.dumps() in python?

I searched in this official document to find difference between the json.dump() and json.dumps() in python. It is clear that they are related with file write option. But what is the detailed difference between them and in what situations one has more advantage than other?

Output: If you want to dump the JSON into a file/socket or whatever, then you should go with dump(). If you only need it as a string (for printing, parsing or whatever) then use dumps() (dump string) As mentioned by Antti Haapala in this answer, there are some minor differences on the ensure_ascii behaviour. This is mostly due to how the underlying write() function works, being that it operates on chunks rather than the whole string. Check his answer for more details on that. json.dump() Serialize obj as a JSON formatted stream to fp (a .write()-supporting file-like object If ensure_ascii is False, some chunks written to fp may be unicode instances json.dumps() Serialize obj to a JSON formatted str If ensure_ascii is False, the result may contain non-ASCII characters and the return value may be a unicode instance


Instruction:
How to re-raise an exception in nested try/except blocks?

I know that if I want to re-raise an exception, I simple use raise without arguments in the respective except block. But given a nested expression like try: something() except SomeError as e: try: plan_B() except AlsoFailsError: raise e # I'd like to raise the SomeError as if plan_B() # didn't raise the AlsoFailsError how can I re-raise the SomeError without breaking the stack trace? raise alone would in this case re-raise the more recent AlsoFailsError. Or how could I refactor my code to avoid this issue?

Output: As of Python 3, the traceback is stored in the exception, so a simple raise e will do the (mostly) right thing: try: something() except SomeError as e: try: plan_B() except AlsoFailsError: raise e # or raise e from None - see below The traceback produced will include an additional notice that SomeError occurred while handling AlsoFailsError (because of raise e being inside except AlsoFailsError). This is misleading because what actually happened is the other way around - we encountered AlsoFailsError, and handled it, while trying to recover from SomeError. To obtain a traceback that doesn't include AlsoFailsError, replace raise e with raise e from None. In Python 2 you'd store the exception type, value, and traceback in local variables and use the three-argument form of raise: try: something() except SomeError: t, v, tb = sys.exc_info() try: plan_B() except AlsoFailsError: raise t, v, tb


Instruction:
Find indices of elements equal to zero in a NumPy array

NumPy has the efficient function/method nonzero() to identify the indices of non-zero elements in an ndarray object. What is the most efficient way to obtain the indices of the elements that do have a value of zero?

Output: numpy.where() is my favorite. >>> x = numpy.array([1,0,2,0,3,0,4,5,6,7,8]) >>> numpy.where(x == 0)[0] array([1, 3, 5]) The method where returns a tuple of ndarrays, each corresponding to a different dimension of the input. Since the input is one-dimensional, the [0] unboxes the tuple's only element.


Instruction:
Parsing XML with namespace in Python via 'ElementTree'

I have the following XML which I want to parse using Python's ElementTree: <rdf:RDF xml:base="http://dbpedia.org/ontology/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:owl="http://www.w3.org/2002/07/owl#" xmlns:xsd="http://www.w3.org/2001/XMLSchema#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#" xmlns="http://dbpedia.org/ontology/"> <owl:Class rdf:about="http://dbpedia.org/ontology/BasketballLeague"> <rdfs:label xml:lang="en">basketball league</rdfs:label> <rdfs:comment xml:lang="en"> a group of sports teams that compete against each other in Basketball </rdfs:comment> </owl:Class> </rdf:RDF> I want to find all owl:Class tags and then extract the value of all rdfs:label instances inside them. I am using the following code: tree = ET.parse("filename") root = tree.getroot() root.findall('owl:Class') Because of the namespace, I am getting the following error. SyntaxError: prefix 'owl' not found in prefix map I tried reading the document at http://effbot.org/zone/element-namespaces.htm but I am still not able to get this working since the above XML has multiple nested namespaces. Kindly let me know how to change the code to find all the owl:Class tags.

Output: You need to give the .find(), findall() and iterfind() methods an explicit namespace dictionary: namespaces = {'owl': 'http://www.w3.org/2002/07/owl#'} # add more as needed root.findall('owl:Class', namespaces) Prefixes are only looked up in the namespaces parameter you pass in. This means you can use any namespace prefix you like; the API splits off the owl: part, looks up the corresponding namespace URL in the namespaces dictionary, then changes the search to look for the XPath expression {http://www.w3.org/2002/07/owl}Class instead. You can use the same syntax yourself too of course: root.findall('{http://www.w3.org/2002/07/owl#}Class') Also see the Parsing XML with Namespaces section of the ElementTree documentation. As of Python 3.8, the ElementTree library also understands the {*} namespace wildcard, so root.findall('{*}Class') would also work (but don't do that if your document can have multiple namespaces that define the Class element). If you can switch to the lxml library things are better; that library supports the same ElementTree API, but collects namespaces for you in .nsmap attribute on elements and generally has superior namespaces support.


Instruction:
If list index exists, do X

I need to code such that if a certain list index exists, then run a function. My failing code: n = int(input("Define number of actors: ")) count = 0 nams = [] while count < n: count = count + 1 print("Define name for actor", count, ":") nams.append(input()) if nams[2]: # 'if nams[2] exists, do something depending on len(nams) if len(nams) > 3: do_something

Output: Could it be more useful for you to use the length of the list len(nams) to inform your decision rather than checking nams[i] for each possible length?


Instruction:
Split a large pandas dataframe

I have a large dataframe with 423244 lines. I want to split this in to 4. I tried the following code which gave an error? ValueError: array split does not result in an equal division for item in np.split(df, 4): print item How to split this dataframe in to 4 groups?

Output: Use np.array_split: Docstring: Split an array into multiple sub-arrays. Please refer to the ``split`` documentation. The only difference between these functions is that ``array_split`` allows `indices_or_sections` to be an integer that does *not* equally divide the axis. In [1]: import pandas as pd In [2]: df = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar', ...: 'foo', 'bar', 'foo', 'foo'], ...: 'B' : ['one', 'one', 'two', 'three', ...: 'two', 'two', 'one', 'three'], ...: 'C' : randn(8), 'D' : randn(8)}) In [3]: print df A B C D 0 foo one -0.174067 -0.608579 1 bar one -0.860386 -1.210518 2 foo two 0.614102 1.689837 3 bar three -0.284792 -1.071160 4 foo two 0.843610 0.803712 5 bar two -1.514722 0.870861 6 foo one 0.131529 -0.968151 7 foo three -1.002946 -0.257468 In [4]: import numpy as np In [5]: np.array_split(df, 3) Out[5]: [ A B C D 0 foo one -0.174067 -0.608579 1 bar one -0.860386 -1.210518 2 foo two 0.614102 1.689837, A B C D 3 bar three -0.284792 -1.071160 4 foo two 0.843610 0.803712 5 bar two -1.514722 0.870861, A B C D 6 foo one 0.131529 -0.968151 7 foo three -1.002946 -0.257468]


Instruction:
GroupBy pandas DataFrame and select most common value

I have a data frame with three string columns. I know that the only one value in the 3rd column is valid for every combination of the first two. To clean the data I have to group by data frame by first two columns and select most common value of the third column for each combination. My code: import pandas as pd from scipy import stats source = pd.DataFrame({ 'Country': ['USA', 'USA', 'Russia', 'USA'], 'City': ['New-York', 'New-York', 'Sankt-Petersburg', 'New-York'], 'Short name': ['NY', 'New', 'Spb', 'NY']}) source.groupby(['Country','City']).agg(lambda x: stats.mode(x['Short name'])[0]) Last line of code doesn't work, it says KeyError: 'Short name' and if I try to group only by City, then I got an AssertionError. What can I do fix it?

Output: You can use value_counts() to get a count series, and get the first row: source.groupby(['Country','City']).agg(lambda x: x.value_counts().index[0]) In case you are wondering about performing other agg functions in the .agg(), try this. # Let's add a new col, "account" source['account'] = [1, 2, 3, 3] source.groupby(['Country','City']).agg( mod=('Short name', lambda x: x.value_counts().index[0]), avg=('account', 'mean'))


Instruction:
Test if lists share any items in python

I want to check if any of the items in one list are present in another list. I can do it simply with the code below, but I suspect there might be a library function to do this. If not, is there a more pythonic method of achieving the same result. In [78]: a = [1, 2, 3, 4, 5] In [79]: b = [8, 7, 6] In [80]: c = [8, 7, 6, 5] In [81]: def lists_overlap(a, b): ....: for i in a: ....: if i in b: ....: return True ....: return False ....: In [82]: lists_overlap(a, b) Out[82]: False In [83]: lists_overlap(a, c) Out[83]: True In [84]: def lists_overlap2(a, b): ....: return len(set(a).intersection(set(b))) > 0 ....:

Output: Short answer: use not set(a).isdisjoint(b), it's generally the fastest. There are four common ways to test if two lists a and b share any items. The first option is to convert both to sets and check their intersection, as such: bool(set(a) & set(b)) Because sets are stored using a hash table in Python, searching them is O(1) (see here for more information about complexity of operators in Python). Theoretically, this is O(n+m) on average for n and m objects in lists a and b. But it must first create sets out of the lists, which can take a non-negligible amount of time, and it supposes that hashing collisions are sparse among your data. The second way to do it is using a generator expression performing iteration on the lists, such as: any(i in a for i in b) This allows to search in-place, so no new memory is allocated for intermediary variables. It also bails out on the first find. But the in operator is always O(n) on lists (see here). Another proposed option is an hybridto iterate through one of the list, convert the other one in a set and test for membership on this set, like so: a = set(a); any(i in a for i in b) A fourth approach is to take advantage of the isdisjoint() method of the (frozen)sets (see here), for example: not set(a).isdisjoint(b) If the elements you search are near the beginning of an array (e.g. it is sorted), the generator expression is favored, as the sets intersection method have to allocate new memory for the intermediary variables: from timeit import timeit >>> timeit('bool(set(a) & set(b))', setup="a=list(range(1000));b=list(range(1000))", number=100000) 26.077727576019242 >>> timeit('any(i in a for i in b)', setup="a=list(range(1000));b=list(range(1000))", number=100000) 0.16220548999262974 Here's a graph of the execution time for this example in function of list size: Note that both axes are logarithmic. This represents the best case for the generator expression. As can be seen, the isdisjoint() method is better for very small list sizes, whereas the generator expression is better for larger list sizes. On the other hand, as the search begins with the beginning for the hybrid and generator expression, if the shared element are systematically at the end of the array (or both lists does not share any values), the disjoint and set intersection approaches are then way faster than the generator expression and the hybrid approach. >>> timeit('any(i in a for i in b)', setup="a=list(range(1000));b=[x+998 for x in range(999,0,-1)]", number=1000)) 13.739536046981812 >>> timeit('bool(set(a) & set(b))', setup="a=list(range(1000));b=[x+998 for x in range(999,0,-1)]", number=1000)) 0.08102107048034668 It is interesting to note that the generator expression is way slower for bigger list sizes. This is only for 1000 repetitions, instead of the 100000 for the previous figure. This setup also approximates well when when no elements are shared, and is the best case for the disjoint and set intersection approaches. Here are two analysis using random numbers (instead of rigging the setup to favor one technique or another): High chance of sharing: elements are randomly taken from [1, 2*len(a)]. Low chance of sharing: elements are randomly taken from [1, 1000*len(a)]. Up to now, this analysis supposed both lists are of the same size. In case of two lists of different sizes, for example a is much smaller, isdisjoint() is always faster: Make sure that the a list is the smaller, otherwise the performance decreases. In this experiment, the a list size was set constant to 5. In summary: If the lists are very small (< 10 elements), not set(a).isdisjoint(b) is always the fastest. If the elements in the lists are sorted or have a regular structure that you can take advantage of, the generator expression any(i in a for i in b) is the fastest on large list sizes; Test the set intersection with not set(a).isdisjoint(b), which is always faster than bool(set(a) & set(b)). The hybrid "iterate through list, test on set" a = set(a); any(i in a for i in b) is generally slower than other methods. The generator expression and the hybrid are much slower than the two other approaches when it comes to lists without sharing elements. In most cases, using the isdisjoint() method is the best approach as the generator expression will take much longer to execute, as it is very inefficient when no elements are shared.


Instruction:
DeprecationWarning: executable_path has been deprecated selenium python

I am using sublime to code python scripts. The following code is for selenium in python to install the driver automatically by using the webdriver_manager package # pip install webdriver-manager from selenium import webdriver from selenium.webdriver.chrome.service import Service from webdriver_manager.chrome import ChromeDriverManager from selenium.webdriver.common.by import By driver = webdriver.Chrome(ChromeDriverManager().install()) driver.maximize_window() #s=Service(path) #driver=webdriver.Chrome(service=s) driver.get('https://www.google.com') driver.find_element(By.NAME, 'q').send_keys('Yasser Khalil') The code works fine but I got a warning like that Demo.py:7: DeprecationWarning: executable_path has been deprecated, please pass in a Service object driver = webdriver.Chrome(ChromeDriverManager().install()) How to fix such a bug?

Output: This error message... DeprecationWarning: executable_path has been deprecated, please pass in a Service object ...implies that the key executable_path will be deprecated in the upcoming releases. This change is inline with the Selenium 4.0 Beta 1 changelog which mentions: Deprecate all but Options and Service arguments in driver instantiation. (#9125,#9128) Solution With selenium4 as the key executable_path is deprecated you have to use an instance of the Service() class along with ChromeDriverManager().install() command as discussed below. Pre-requisites Ensure that: Selenium is upgraded to v4.0.0 pip3 install -U selenium Webdriver Manager for Python is installed pip3 install webdriver-manager You can find a detailed discussion on installing Webdriver Manager for Python in ModuleNotFoundError: No module named 'webdriver_manager' error even after installing webdrivermanager Selenium v4 compatible Code Block from selenium import webdriver from selenium.webdriver.chrome.service import Service from webdriver_manager.chrome import ChromeDriverManager driver = webdriver.Chrome(service=Service(ChromeDriverManager().install())) driver.get("https://www.google.com") Console Output: [WDM] - ====== WebDriver manager ====== [WDM] - Current google-chrome version is 96.0.4664 [WDM] - Get LATEST driver version for 96.0.4664 [WDM] - Driver [C:\Users\Admin\.wdm\drivers\chromedriver\win32\96.0.4664.45\chromedriver.exe] found in cache You can find a detailed discussion on installing Webdriver Manager for Python in Selenium ChromeDriver issue using Webdriver Manager for Python Incase you want to pass the Options() object you can use: from selenium import webdriver from selenium.webdriver.chrome.options import Options from selenium.webdriver.chrome.service import Service from webdriver_manager.chrome import ChromeDriverManager options = Options() options.add_argument("start-maximized") driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options) driver.get("https://www.google.com") TL; DR You can find the relevant Bug Report/Pull Request in: Bug Report: deprecate all but Options and Service arguments in driver instantiation Pull Request: deprecate all but Options and Service arguments in driver instantiation


Instruction:
Multiprocessing - Pipe vs Queue

What are the fundamental differences between queues and pipes in Python's multiprocessing package? In what scenarios should one choose one over the other? When is it advantageous to use Pipe()? When is it advantageous to use Queue()?

Output: What are the fundamental differences between queues and pipes in Python's multiprocessing package? Major Edit of this answer (CY2024): concurrency As of modern python versions if you don't need your producers and consumers to communicate, that's the only real use-case for python multiprocessing. If you only need python concurrency, use concurrent.futures. This example uses concurrent.futures to make four calls to do_something_slow(), which has a one-second delay. If your machine has at least four cores, running this four-second-aggregate series of function calls only takes one-second. By default, concurrent.futures spawns workers corresponding to the number of CPU cores you have. import concurrent.futures import time def do_slow_thing(input_str: str) -> str: """Return modified input string after a 1-second delay""" if isinstance(input_str, str): time.sleep(1) return "1-SECOND-DELAY " + input_str else: return "INPUT ERROR" if __name__=="__main__": # Define some inputs for process pool all_inputs = [ "do", "foo", "moo", "chew", ] # Spawn a process pool with the default number of workers... with concurrent.futures.ProcessPoolExecutor(max_workers=None) as executor: # For each string in all_inputs, call do_slow_thing() # in parallel across the process worker pool these_futures = [executor.submit(do_slow_thing, ii) for ii in all_inputs] # Wait for all processes to finish concurrent.futures.wait(these_futures) # Get the results from the process pool execution... each # future.result() call is the return value from do_slow_thing() string_outputs = [future.result() for future in these_futures] for tmp in string_outputs: print(tmp) With at least four CPU cores, you'll see this printed after roughly one-second... $ time python stackoverflow.py 1-SECOND-DELAY do 1-SECOND-DELAY foo 1-SECOND-DELAY moo 1-SECOND-DELAY chew real 0m1.058s user 0m0.060s sys 0m0.017s $ Original Answer At this point, the only major use-case for multiprocessing is to facilitate your producers and consumers talking to each other during execution. Most people don't need that. However, if you want communication via queue / pipes, you can find my original answer to the OP's question below (which profiles how fast they are). The existing comments on this answer refer to the aforementioned answer below


Instruction:
How can I iterate over overlapping (current, next) pairs of values from a list?

I sometimes need to iterate a list in Python looking at the "current" element and the "next" element. I have, till now, done so with code like: for current, next in zip(the_list, the_list[1:]): # Do something This works and does what I expect, but is there's a more idiomatic or efficient way to do the same thing? Some answers to this problem can simplify by addressing the specific case of taking only two elements at a time. For the general case of N elements at a time, see Rolling or sliding window iterator?.

Output: The documentation for 3.8 provides this recipe: import itertools def pairwise(iterable): "s -> (s0, s1), (s1, s2), (s2, s3), ..." a, b = itertools.tee(iterable) next(b, None) return zip(a, b) For Python 2, use itertools.izip instead of zip to get the same kind of lazy iterator (zip will instead create a list): import itertools def pairwise(iterable): "s -> (s0, s1), (s1, s2), (s2, s3), ..." a, b = itertools.tee(iterable) next(b, None) return itertools.izip(a, b) How this works: First, two parallel iterators, a and b are created (the tee() call), both pointing to the first element of the original iterable. The second iterator, b is moved 1 step forward (the next(b, None)) call). At this point a points to s0 and b points to s1. Both a and b can traverse the original iterator independently - the izip function takes the two iterators and makes pairs of the returned elements, advancing both iterators at the same pace. Since tee() can take an n parameter (the number of iterators to produce), the same technique can be adapted to produce a larger "window". For example: def threes(iterator): "s -> (s0, s1, s2), (s1, s2, s3), (s2, s3, 4), ..." a, b, c = itertools.tee(iterator, 3) next(b, None) next(c, None) next(c, None) return zip(a, b, c) Caveat: If one of the iterators produced by tee advances further than the others, then the implementation needs to keep the consumed elements in memory until every iterator has consumed them (it cannot 'rewind' the original iterator). Here it doesn't matter because one iterator is only 1 step ahead of the other, but in general it's easy to use a lot of memory this way.


Instruction:
Python if not == vs if !=

What is the difference between these two lines of code: if not x == 'val': and if x != 'val': Is one more efficient than the other? Would it be better to use if x == 'val': pass else:

Output: Using dis to look at the bytecode generated for the two versions: not == 4 0 LOAD_FAST 0 (foo) 3 LOAD_FAST 1 (bar) 6 COMPARE_OP 2 (==) 9 UNARY_NOT 10 RETURN_VALUE != 4 0 LOAD_FAST 0 (foo) 3 LOAD_FAST 1 (bar) 6 COMPARE_OP 3 (!=) 9 RETURN_VALUE The latter has fewer operations, and is therefore likely to be slightly more efficient. It was pointed out in the commments (thanks, @Quincunx) that where you have if foo != bar vs. if not foo == bar the number of operations is exactly the same, it's just that the COMPARE_OP changes and POP_JUMP_IF_TRUE switches to POP_JUMP_IF_FALSE: not == : 2 0 LOAD_FAST 0 (foo) 3 LOAD_FAST 1 (bar) 6 COMPARE_OP 2 (==) 9 POP_JUMP_IF_TRUE 16 != 2 0 LOAD_FAST 0 (foo) 3 LOAD_FAST 1 (bar) 6 COMPARE_OP 3 (!=) 9 POP_JUMP_IF_FALSE 16 In this case, unless there was a difference in the amount of work required for each comparison, it's unlikely you'd see any performance difference at all. However, note that the two versions won't always be logically identical, as it will depend on the implementations of __eq__ and __ne__ for the objects in question. Per the data model documentation: There are no implied relationships among the comparison operators. The truth of x==y does not imply that x!=y is false. For example: >>> class Dummy(object): def __eq__(self, other): return True def __ne__(self, other): return True >>> not Dummy() == Dummy() False >>> Dummy() != Dummy() True Finally, and perhaps most importantly: in general, where the two are logically identical, x != y is much more readable than not x == y.


Instruction:
Create a single executable from a Python project

I want to create a single executable from my Python project. A user should be able to download and run it without needing Python installed. If I were just distributing a package, I could use pip, wheel, and PyPI to build and distribute it, but this requires that the user has Python and knows how to install packages. What can I use to build a self-contained executable from a Python project?

Output: There are several different ways of doing this. The first -- and likely most common -- way is to use "freeze" style programs. These programs work by bundling together Python and your program, essentially combining them into a single executable: PyInstaller: Website || Repo || PyPi Supports Python 3.7 - 3.10 on Windows, Mac, and Linux. cx_Freeze: Website || Repo || PyPi Supports Python 3.6 - 3.10 on Windows, Mac, and Linux. py2exe: Website || Repo || PyPi Supports Python 3.7 - 3.10 on Windows only. py2app: Website || Repo || PyPi Supports Python 3.6 - 3.10 on Macs only. The main thing to keep in mind is that these types of programs will generally only produce an exe for the operating system you run it in. So for example, running Pyinstaller in Windows will produce a Windows exe, but running Pyinstaller in Linux will produce a Linux exe. If you want to produce an exe for multiple operating systems, you will have to look into using virtual machines or something like Wine. Of course, that's not the only way of doing things: pynsist: Website || Repo || PyPi Pynsist will create a Windows installer for your program which will directly install Python on the user's computer instead of bundling it with your code and create shortcuts that link to your Python script. The pynsist tool itself requires Python 3.5+ to run, but supports bundling any version of Python with your program. Pynsist will create Windows installers only, but can be run from Windows, Mac, and Linux. See their FAQ for more details. Nuitka: Website || Repo (Github mirror) || PyPi Nuitka will literally compile your Python code and produce an exe (as opposed to the other projects, which simply include Python) to try and speed up your code. As a side effect, you'll also get a handy exe you can distribute. Note that you need to have a C++ compiler available on your system. Supports Python 2.6 - 2.7 and Python 3.3 - 3.10 on Windows, Mac, and Linux. cython: Website || Repo || PyPi Cython is similar to Nuitka in that it is a Python compiler. However, instead of directly compiling your code, it'll compile it to C. You can then take that C code and turn your code into an exe. You'll need to have a C compiler available on your system. Supports Python 2.7 and Python 3.3 - 3.11 on Windows, Mac, and Linux. My personal preference is to use PyInstaller since it was the easiest for me to get up and running, was designed to work nicely with various popular libraries such as numpy or pygame, and has great compatibility with various OSes and Python versions. However, I've also successfully built various exes using cx_Freeze without too much difficulty, so you should also consider trying that program out. I haven't yet had a chance to to try pynist, Nuitka, or Cython extensively, but they seem like pretty interesting and innovative solutions. If you run into trouble using the first group of programs, it might be worthwhile to try one of these three. Since they work fundamentally differently then the Pyinstaller/cx_freeze-style programs, they might succeed in those odd edge cases where the first group fails. In particular, I think pynist is a good way of sidestepping the entire issue of distributing your code altogether: Macs and Linux already have native support for Python, and just installing Python on Windows might genuinely be the cleanest solution. (The downside is now that you need to worry about targeting multiple versions of Python + installing libraries). Nuitka and Cython (in my limited experience) seem to work fairly well. Again, I haven't tested them extensively myself, and so my main observation is that they seem to take much longer to produce an exe then the "freeze" style programs do. All this being said, converting your Python program into an executable isn't necessarily the only way of distributing your code. To learn more about what other options are available, see the following links: https://packaging.python.org/overview/#packaging-python-applications https://docs.python-guide.org/shipping/packaging/#for-linux-distributions


Instruction:
Does it make sense to use Conda + Poetry?

Does it make sense to use Conda + Poetry for a Machine Learning project? Allow me to share my (novice) understanding and please correct or enlighten me: As far as I understand, Conda and Poetry have different purposes but are largely redundant: Conda is primarily a environment manager (in fact not necessarily Python), but it can also manage packages and dependencies. Poetry is primarily a Python package manager (say, an upgrade of pip), but it can also create and manage Python environments (say, an upgrade of Pyenv). My idea is to use both and compartmentalize their roles: let Conda be the environment manager and Poetry the package manager. My reasoning is that (it sounds like) Conda is best for managing environments and can be used for compiling and installing non-python packages, especially CUDA drivers (for GPU capability), while Poetry is more powerful than Conda as a Python package manager. I've managed to make this work fairly easily by using Poetry within a Conda environment. The trick is to not use Poetry to manage the Python environment: I'm not using commands like poetry shell or poetry run, only poetry init, poetry install etc (after activating the Conda environment). For full disclosure, my environment.yml file (for Conda) looks like this: name: N channels: - defaults - conda-forge dependencies: - python=3.9 - cudatoolkit - cudnn and my poetry.toml file looks like that: [tool.poetry] name = "N" authors = ["B"] [tool.poetry.dependencies] python = "3.9" torch = "^1.10.1" [build-system] requires = ["poetry-core>=1.0.0"] build-backend = "poetry.core.masonry.api" To be honest, one of the reasons I proceeded this way is that I was struggling to install CUDA (for GPU support) without Conda. Does this project design look reasonable to you?

Output: 2024-04-05 update: It looks like my tips proved to be useful to many people, but they are not needed anymore. Just use Pixi. It's still alpha, but it works great, and provides the features of the Conda + Poetry setup in a simpler and more unified way. In particular, Pixi supports: installing packages both from Conda channels and from PyPi, lockfiles, creating multiple features and environments (prod, dev, etc.), very efficient package version resolution, not just faster than Conda (which is very slow), but in my experience also faster than Mamba, Poetry and pip. Making a Pixi env look like a Conda env One non-obvious tip about Pixi is that you can easily make your project's Pixi environment visible as a Conda environment, which may be useful e.g. in VS Code, which allows choosing Python interpreters and Jupyter kernels from detected Conda environments. All you need to do is something like: ln -s /path/to/my/project/.pixi/envs/default /path/to/conda/base/envs/conda-name-of-my-env The first path is the path to your Pixi environment, which resides in your project directory, under .pixi/envs, and the second path needs to be within one of Conda's environment directories, which can be found with conda config --show envs_dirs. Original answer: I have experience with a Conda + Poetry setup, and it's been working fine. The great majority of my dependencies are specified in pyproject.toml, but when there's something that's unavailable in PyPI, or installing it with Conda is easier, I add it to environment.yml. Moreover, Conda is used as a virtual environment manager, which works well with Poetry: there is no need to use poetry run or poetry shell, it is enough to activate the right Conda environment. Tips for creating a reproducible environment Add Poetry, possibly with a version number (if needed), as a dependency in environment.yml, so that you get Poetry installed when you run conda create, along with Python and other non-PyPI dependencies. Add conda-lock, which gives you lock files for Conda dependencies, just like you have poetry.lock for Poetry dependencies. Consider using mamba which is generally compatible with conda, but is better at resolving conflicts, and is also much faster. An additional benefit is that all users of your setup will use the same package resolver, independent from the locally-installed version of Conda. By default, use Poetry for adding Python dependencies. Install packages via Conda if there's a reason to do so (e.g. in order to get a CUDA-enabled version). In such a case, it is best to specify the package's exact version in environment.yml, and after it's installed, to add an entry with the same version specification to Poetry's pyproject.toml (without ^ or ~ before the version number). This will let Poetry know that the package is there and should not be upgraded. If you use a different channels that provide the same packages, it might be not obvious which channel a particular package will be downloaded from. One solution is to specify the channel for the package using the :: notation (see the pytorch entry below), and another solution is to enable strict channel priority. Unfortunately, in Conda 4.x there is no way to enable this option through environment.yml. Note that Python adds user site-packages to sys.path, which may cause lack of reproducibility if the user has installed Python packages outside Conda environments. One possible solution is to make sure that the PYTHONNOUSERSITE environment variable is set to True (or to any other non-empty value). Example environment.yml: name: my_project_env channels: - pytorch - conda-forge # We want to have a reproducible setup, so we don't want default channels, # which may be different for different users. All required channels should # be listed explicitly here. - nodefaults dependencies: - python=3.10.* # or don't specify the version and use the latest stable Python - mamba - pip # pip must be mentioned explicitly, or conda-lock will fail - poetry=1.* # or 1.1.*, or no version at all -- as you want - tensorflow=2.8.0 - pytorch::pytorch=1.11.0 - pytorch::torchaudio=0.11.0 - pytorch::torchvision=0.12.0 # Non-standard section listing target platforms for conda-lock: platforms: - linux-64 virtual-packages.yml (may be used e.g. when we want conda-lock to generate CUDA-enabled lock files even on platforms without CUDA): subdirs: linux-64: packages: __cuda: 11.5 First-time setup You can avoid playing with the bootstrap env and simplify the example below if you have conda-lock, mamba and poetry already installed outside your target environment. # Create a bootstrap env conda create -p /tmp/bootstrap -c conda-forge mamba conda-lock poetry='1.*' conda activate /tmp/bootstrap # Create Conda lock file(s) from environment.yml conda-lock -k explicit --conda mamba # Set up Poetry poetry init --python=~3.10 # version spec should match the one from environment.yml # Fix package versions installed by Conda to prevent upgrades poetry add --lock tensorflow=2.8.0 torch=1.11.0 torchaudio=0.11.0 torchvision=0.12.0 # Add conda-lock (and other packages, as needed) to pyproject.toml and poetry.lock poetry add --lock conda-lock # Remove the bootstrap env conda deactivate rm -rf /tmp/bootstrap # Add Conda spec and lock files git add environment.yml virtual-packages.yml conda-linux-64.lock # Add Poetry spec and lock files git add pyproject.toml poetry.lock git commit Usage The above setup may seem complex, but it can be used in a fairly simple way. Creating the environment conda create --name my_project_env --file conda-linux-64.lock conda activate my_project_env poetry install Activating the environment conda activate my_project_env Updating the environment # Re-generate Conda lock file(s) based on environment.yml conda-lock -k explicit --conda mamba # Update Conda packages based on re-generated lock file mamba update --file conda-linux-64.lock # Update Poetry packages and re-generate poetry.lock poetry update


Instruction:
What does a leading . (dot, period) in an import statement in Python mean?

I'm looking over the code for Python's multiprocessing module, and it contains this line: from ._multiprocessing import win32, Connection, PipeConnection instead of from _multiprocessing import win32, Connection, PipeConnection the subtle difference being the period before _multiprocessing. What does that mean? Why the period?

Output: That's the syntax for explicit relative imports. It means import from the current package.


Instruction:
Convert from ASCII string encoded in Hex to plain ASCII?

How can I convert from hex to plain ASCII in Python? Note that, for example, I want to convert "0x7061756c" to "paul".

Output: A slightly simpler solution (python 2 only): >>> "7061756c".decode("hex") 'paul'


Instruction:
How to run functions in parallel?

I am trying to run multiple functions in parallel in Python. I have something like this: files.py import common #common is a util class that handles all the IO stuff dir1 = 'C:\folder1' dir2 = 'C:\folder2' filename = 'test.txt' addFiles = [25, 5, 15, 35, 45, 25, 5, 15, 35, 45] def func1(): c = common.Common() for i in range(len(addFiles)): c.createFiles(addFiles[i], filename, dir1) c.getFiles(dir1) time.sleep(10) c.removeFiles(addFiles[i], dir1) c.getFiles(dir1) def func2(): c = common.Common() for i in range(len(addFiles)): c.createFiles(addFiles[i], filename, dir2) c.getFiles(dir2) time.sleep(10) c.removeFiles(addFiles[i], dir2) c.getFiles(dir2) I want to call func1 and func2 and have them run at the same time. The functions do not interact with each other or on the same object. Right now I have to wait for func1 to finish before func2 to start. How do I do something like below: process.py from files import func1, func2 runBothFunc(func1(), func2()) I want to be able to create both directories pretty close to the same time because every min I am counting how many files are being created. If the directory isn't there it will throw off my timing.

Output: You could use threading or multiprocessing. Due to peculiarities of CPython, threading is unlikely to achieve true parallelism. For this reason, multiprocessing is generally a better bet. Here is a complete example: from multiprocessing import Process def func1(): print("func1: starting") for i in range(10000000): pass print("func1: finishing") def func2(): print("func2: starting") for i in range(10000000): pass print("func2: finishing") if __name__ == "__main__": p1 = Process(target=func1) p1.start() p2 = Process(target=func2) p2.start() p1.join() p2.join() The mechanics of starting/joining child processes can easily be encapsulated into a function along the lines of your runBothFunc: def runInParallel(*fns): proc = [] for fn in fns: p = Process(target=fn) p.start() proc.append(p) for p in proc: p.join() runInParallel(func1, func2)


Instruction:
Why did Flask start failing with "ImportError: cannot import name 'url_quote' from 'werkzeug.urls'"?

Environment: Python 3.10.11 Flask==2.2.2 I run my Flask backend code in docker container, with BASE Image: FROM pytorch/pytorch:2.0.1-cuda11.7-cudnn8-runtime But when I run the pytest with version pytest 7.4.2, pip install pytest pytest it raised an Error, with logs: ==================================== ERRORS ==================================== _____________ ERROR collecting tests/test_fiftyone_utils_utils.py ______________ ImportError while importing test module '/builds/kw/data-auto-analysis-toolkit-backend/tests/test_fiftyone_utils_utils.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /opt/conda/lib/python3.10/importlib/__init__.py:126: in import_module return _bootstrap._gcd_import(name[level:], package, level) tests/test_fiftyone_utils_utils.py:2: in <module> import daat # noqa: F401 /opt/conda/lib/python3.10/site-packages/daat-1.0.0-py3.10.egg/daat/__init__.py:1: in <module> from daat.app import app /opt/conda/lib/python3.10/site-packages/daat-1.0.0-py3.10.egg/daat/app/__init__.py:6: in <module> from flask import Flask, jsonify, request /opt/conda/lib/python3.10/site-packages/flask/__init__.py:5: in <module> from .app import Flask as Flask /opt/conda/lib/python3.10/site-packages/flask/app.py:30: in <module> from werkzeug.urls import url_quote E ImportError: cannot import name 'url_quote' from 'werkzeug.urls' (/opt/conda/lib/python3.10/site-packages/werkzeug/urls.py) My codes works well when I directly run it with python run.py run.py shown below from daat import app app.run(host='0.0.0.0') I guess it should be the pytest versions issue, because it used to work well without changing any related code, and I use pip install pytest without defined a specific version. And my backend runs well without pytest.

Output: I had the same problem. It is because Werkzeug 3.0.0 was released and Flask doesn't specify the dependency correctly (requirements says Werkzeug>=2.2.0). This is why, Werkzeug 3.0.0 is still installed and Flask 2.2.2 isn't made for Werkzeug 3.0.0. Solution: Just set a fix version for Werkzeug such as Werkzeug==2.2.2 in your requirements.txt and it should work.


Instruction:
How do I create a list of random numbers without duplicates?

I tried using random.randint(0, 100), but some numbers were the same. Is there a method/module to create a list unique random numbers?

Output: This will return a list of 10 numbers selected from the range 0 to 99, without duplicates. import random random.sample(range(100), 10)


Instruction:
How can I read a function's signature including default argument values?

Given a function object, how can I get its signature? For example, for: def my_method(first, second, third='something'): pass I would like to get "my_method(first, second, third='something')".

Output: import inspect def foo(a, b, x='blah'): pass print(inspect.signature(foo)) # (a, b, x='blah') Python 3.5+ recommends inspect.signature().


Instruction:
Seaborn showing scientific notation in heatmap for 3-digit numbers

I'm creating a heatmap from a pandas pivot_table as below: table2 = pd.pivot_table(df,values='control',columns='Year',index='Region',aggfunc=np.sum) sns.heatmap(table2,annot=True,cmap='Blues') It creates a heat map as shown below. You can see the numbers are not huge (max 750), but it's showing them in scientific notation. If I view the table itself this is not the case. Any idea on how I could get it to show the numbers in plain notation?

Output: According to the docs, the param fmt='.2g' is being applied because you've set annot=True so you can modify the format being applied to: sns.heatmap(table2,annot=True,cmap='Blues', fmt='g')


Instruction:
How can I put multiple statements in one line?

I know a little bit of comprehensions in Python, but they seem very hard to 'read'. The way I see it, a comprehension might accomplish the same as the following code: for i in range(10): if i == 9: print('i equals 9') This code is much easier to read than how comprehensions currently work, but I've noticed you can’t have two :s in one line. This brings me to: Is there a way I can get the following example into one line? try: if sam[0] != 'harry': print('hello', sam) except: pass Something like this would be great: try: if sam[0] != 'harry': print('hellp', sam) except:pass But again I encounter the conflicting :s. I'd also love to know if there's a way to run try (or something like it) without except. It seems entirely pointless that I need to put except:pass in there. It’s a wasted line.

Output: Unfortunately, what you want is not possible with Python (which makes Python close to useless for command-line one-liner programs). Even explicit use of parentheses does not avoid the syntax exception. You can get away with a sequence of simple statements, separated by semicolon: for i in range(10): print "foo"; print "bar" But as soon as you add a construct that introduces an indented block (like if), you need the line break. Also, for i in range(10): print "i equals 9" if i==9 else None is legal and might approximate what you want. If you are still determined to use one-liners, see eprovst's answer. As for the try ... except thing: It would be totally useless without the except. try says "I want to run this code, but it might throw an exception". If you don't care about the exception, leave out the try. But as soon as you put it in, you're saying "I want to handle a potential exception". The pass then says you wish to not handle it specifically. But that means your code will continue running, which it wouldn't otherwise.


Instruction:
how to set "camera position" for 3d plots using python/matplotlib?

I'm learning how to use mplot3d to produce nice plots of 3d data and I'm pretty happy so far. What I am trying to do at the moment is a little animation of a rotating surface. For that purpose, I need to set a camera position for the 3D projection. I guess this must be possible since a surface can be rotated using the mouse when using matplotlib interactively. But how can I do this from a script? I found a lot of transforms in mpl_toolkits.mplot3d.proj3d but I could not find out how to use these for my purpose and I didn't find any example for what I'm trying to do.

Output: By "camera position," it sounds like you want to adjust the elevation and the azimuth angle that you use to view the 3D plot. You can set this with ax.view_init. I've used the below script to first create the plot, then I determined a good elevation, or elev, from which to view my plot. I then adjusted the azimuth angle, or azim, to vary the full 360deg around my plot, saving the figure at each instance (and noting which azimuth angle as I saved the plot). For a more complicated camera pan, you can adjust both the elevation and angle to achieve the desired effect. from mpl_toolkits.mplot3d import Axes3D ax = Axes3D(fig) ax.scatter(xx,yy,zz, marker='o', s=20, c="goldenrod", alpha=0.6) for ii in xrange(0,360,1): ax.view_init(elev=10., azim=ii) savefig("movie%d.png" % ii)


Instruction:
How to merge multiple dataframes

I have different dataframes and need to merge them together based on the date column. If I only had two dataframes, I could use df1.merge(df2, on='date'), to do it with three dataframes, I use df1.merge(df2.merge(df3, on='date'), on='date'), however it becomes really complex and unreadable to do it with multiple dataframes. All dataframes have one column in common -date, but they don't have the same number of rows nor columns and I only need those rows in which each date is common to every dataframe. So, I'm trying to write a recursion function that returns a dataframe with all data but it didn't work. How should I merge multiple dataframes then? I tried different ways and got errors like out of range, keyerror 0/1/2/3 and can not merge DataFrame with instance of type <class 'NoneType'>. This is the script I wrote: dfs = [df1, df2, df3] # list of dataframes def mergefiles(dfs, countfiles, i=0): if i == (countfiles - 2): # it gets to the second to last and merges it with the last return dfm = dfs[i].merge(mergefiles(dfs[i+1], countfiles, i=i+1), on='date') return dfm print(mergefiles(dfs, len(dfs))) An example: df_1: May 19, 2017;1,200.00;0.1% May 18, 2017;1,100.00;0.1% May 17, 2017;1,000.00;0.1% May 15, 2017;1,901.00;0.1% df_2: May 20, 2017;2,200.00;1000000;0.2% May 18, 2017;2,100.00;1590000;0.2% May 16, 2017;2,000.00;1230000;0.2% May 15, 2017;2,902.00;1000000;0.2% df_3: May 21, 2017;3,200.00;2000000;0.3% May 17, 2017;3,100.00;2590000;0.3% May 16, 2017;3,000.00;2230000;0.3% May 15, 2017;3,903.00;2000000;0.3% Expected merge result: May 15, 2017; 1,901.00;0.1%; 2,902.00;1000000;0.2%; 3,903.00;2000000;0.3%

Output: Short answer df_merged = reduce(lambda left,right: pd.merge(left,right,on=['DATE'], how='outer'), data_frames) Long answer Below, is the most clean, comprehensible way of merging multiple dataframe if complex queries aren't involved. Just simply merge with DATE as the index and merge using OUTER method (to get all the data). import pandas as pd from functools import reduce df1 = pd.read_table('file1.csv', sep=',') df2 = pd.read_table('file2.csv', sep=',') df3 = pd.read_table('file3.csv', sep=',') Now, basically load all the files you have as data frame into a list. And, then merge the files using merge or reduce function. # compile the list of dataframes you want to merge data_frames = [df1, df2, df3] Note: you can add as many data-frames inside the above list. This is the good part about this method. No complex queries involved. To keep the values that belong to the same date you need to merge it on the DATE df_merged = reduce(lambda left,right: pd.merge(left,right,on=['DATE'], how='outer'), data_frames) # if you want to fill the values that don't exist in the lines of merged dataframe simply fill with required strings as df_merged = reduce(lambda left,right: pd.merge(left,right,on=['DATE'], how='outer'), data_frames).fillna('void') Now, the output will the values from the same date on the same lines. You can fill the non existing data from different frames for different columns using fillna(). Then write the merged data to the csv file if desired. pd.DataFrame.to_csv(df_merged, 'merged.txt', sep=',', na_rep='.', index=False) This should give you DATE VALUE1 VALUE2 VALUE3 ....


Instruction:
Choosing a file in Python with simple Dialog

I would like to get file path as input in my Python console application. Currently I can only ask for full path as an input in the console. Is there a way to trigger a simple user interface where users can select file instead of typing the full path?

Output: How about using tkinter? from tkinter import Tk from tkinter.filedialog import askopenfilename Tk().withdraw() # we don't want a full GUI, so keep the root window from appearing filename = askopenfilename() # show an "Open" dialog box and return the path to the selected file print(filename) Done!


Instruction:
Adding headers to the Requests module

Earlier, I used the httplib module to add a header in the request. Now I am trying the same thing with the Requests module. This is the Python Requests module I am using: requests 2.32.5 How can I add a header to request.post() and request.get()? Say, I have to add a foobar key in each request in the header.

Output: From Requests' Quickstart: url = 'https://api.github.com/some/endpoint' payload = {'some': 'data'} headers = {'content-type': 'application/json'} r = requests.post(url, data=json.dumps(payload), headers=headers) You just need to create a dict with your headers (key: value pairs where the key is the name of the header and the value is, well, the value of the pair) and pass that dict to the headers parameter on the .get or .post method. So more specific to your question: headers = {'foobar': 'raboof'} requests.get('http://himom.com', headers=headers)


Instruction:
How to run headless Firefox with Selenium in Python?

I am running this code with python, selenium, and firefox but still get 'head' version of firefox: binary = FirefoxBinary('C:\\Program Files (x86)\\Mozilla Firefox\\firefox.exe', log_file=sys.stdout) binary.add_command_line_options('-headless') self.driver = webdriver.Firefox(firefox_binary=binary) I also tried some variations of binary: binary = FirefoxBinary('C:\\Program Files\\Nightly\\firefox.exe', log_file=sys.stdout) binary.add_command_line_options("--headless")

Output: To invoke Firefox Browser headlessly, you can set the headless property through Options() class as follows: from selenium import webdriver from selenium.webdriver.firefox.options import Options options = Options() options.add_argument("--headless") driver = webdriver.Firefox(options=options) driver.get("http://google.com/") print("Headless Firefox Initialized") driver.quit() There's another way to accomplish headless mode. If you need to disable or enable the headless mode in Firefox, without changing the code, you can set the environment variable MOZ_HEADLESS to whatever if you want Firefox to run headless, or don't set it at all. This is very useful when you are using for example continuous integration and you want to run the functional tests in the server but still be able to run the tests in normal mode in your PC. $ MOZ_HEADLESS=1 python manage.py test # testing example in Django with headless Firefox or $ export MOZ_HEADLESS=1 # this way you only have to set it once $ python manage.py test functional/tests/directory $ unset MOZ_HEADLESS # if you want to disable headless mode


Instruction:
How exactly do Django content types work?

I'm really having a difficult time grasping the concept of Django's content types. It feels very hackish and, ultimately, against how Python tends to do things. That being said, if I'm going to use Django then I have to work within the confines of the framework. So I'm coming here wondering if anyone can give a practical real world example of how a content type works and how you would implement it. Almost all the tutorials (mostly on blogs) I have reviewed don't do a great job really covering the concept. They seem to pick up where the Django documentation left off (what seems like nowhere).

Output: So you want to use the Content Types framework on your work? Start by asking yourself this question: "Do any of these models need to be related in the same way to other models and/or will I be reusing these relationships in unforseen ways later down the road?" The reason why we ask this question is because this is what the Content Types framework does best: it creates generic relations between models. Blah blah, let's dive into some code and see what I mean. # ourapp.models from django.conf import settings from django.db import models # Assign the User model in case it has been "swapped" User = settings.AUTH_USER_MODEL # Create your models here class Post(models.Model): author = models.ForeignKey(User) title = models.CharField(max_length=75) slug = models.SlugField(unique=True) body = models.TextField(blank=True) class Picture(models.Model): author = models.ForeignKey(User) image = models.ImageField() caption = models.TextField(blank=True) class Comment(models.Model): author = models.ForeignKey(User) body = models.TextField(blank=True) post = models.ForeignKey(Post) picture = models.ForeignKey(Picture) Okay, so we do have a way to theoretically create this relationship. However, as a Python programmer, your superior intellect is telling you this sucks and you can do better. High five! Enter the Content Types framework! Well, now we're going to take a close look at our models and rework them to be more "reusable" and intuitive. Let's start by getting rid of the two foreign keys on our Comment model and replace them with a GenericForeignKey. # ourapp.models from django.contrib.contenttypes.fields import GenericForeignKey from django.contrib.contenttypes.models import ContentType ... class Comment(models.Model): author = models.ForeignKey(User) body = models.TextField(blank=True) content_type = models.ForeignKey(ContentType) object_id = models.PositiveIntegerField() content_object = GenericForeignKey() So, what happened? Well, we went in and added the necessary code to allow for a generic relation to other models. Notice how there is more than just a GenericForeignKey, but also a ForeignKey to ContentType and a PositiveIntegerField for the object_id. These fields are for telling Django what type of object this is related to and what the id is for that object. In reality, this makes sense because Django will need both to lookup these related objects. Well, that's not very Python-like... it’s kinda ugly! You are probably looking for air-tight, spotless, intuitive code that would make Guido van Rossum proud. I get you. Let's look at the GenericRelation field so we can put a pretty bow on this. # ourapp.models from django.contrib.contenttypes.fields import GenericRelation ... class Post(models.Model): author = models.ForeignKey(User) title = models.CharField(max_length=75) slug = models.SlugField(unique=True) body = models.TextField(blank=True) comments = GenericRelation('Comment') class Picture(models.Model): author = models.ForeignKey(User) image = models.ImageField() caption = models.TextField(blank=True) comments = GenericRelation('Comment') Bam! Just like that you can work with the Comments for these two models. In fact, let's go ahead and do that in our shell (type python manage.py shell from your Django project directory). >>> from django.contrib.auth import get_user_model >>> from ourapp.models import Picture, Post # We use get_user_model() since we are referencing directly User = get_user_model() # Grab our own User object >>> me = User.objects.get(username='myusername') # Grab the first of our own pictures so we can comment on it >>> pic = Picture.objects.get(author=me) # Let's start making a comment for our own picture >>> pic.comments.create(author=me, body="Man, I'm cool!") # Let's go ahead and retrieve the comments for this picture now >>> pic.comments.all() [<Comment: "Man, I'm cool!">] # Same for Post comments >>> post = Post.objects.get(author=me) >>> post.comments.create(author=me, body="So easy to comment now!") >>> post.comments.all() [<Comment: "So easy to comment now!"] It's that simple. What are the other practical implications of these "generic" relations? Generic foreign keys allow for less intrusive relations between various applications. For example, let's say we pulled the Comment model out into its own app named chatterly. Now we want to create another application named noise_nimbus where people store their music to share with others. What if we want to add comments to those songs? Well, we can just draw a generic relation: # noise_nimbus.models from django.conf import settings from django.contrib.contenttypes.fields import GenericRelation from django.db import models from chatterly.models import Comment # For a third time, we take the time to ensure custom Auth isn't overlooked User = settings.AUTH_USER_MODEL # Create your models here class Song(models.Model): ''' A song which can be commented on. ''' file = models.FileField() author = models.ForeignKey(User) title = models.CharField(max_length=75) slug = models.SlugField(unique=True) description = models.TextField(blank=True) comments = GenericRelation(Comment) I hope you guys found this helpful as I would have loved to have come across something that showed me the more realistic application of GenericForeignKey and GenericRelation fields. Is this too good to be true? As with anything in life, there are pros and cons. Anytime you add more code and more abstraction, the underlying processes becomes heavier and a bit slower. Adding generic relations can add a little bit of a performance dampener despite the fact it will try and smart cache its results. All in all, it comes down to whether the cleanliness and simplicity outweighs the small performance costs. For me, the answer is a million times yes. There is more to the Content Types framework than I have displayed here. There is a whole level of granularity and more verbose usage, but for the average individual, this is how you will be using it 9 out of 10 times in my opinion. Generic relationizers(?) beware! A rather large caveat is that when you use a GenericRelation, if the model which has the GenericRelation applied (Picture) is deleted, all related (Comment) objects will also be deleted. Or at least as of the time of this writing.


Instruction:
Replace NaN in one column with value from corresponding row of second column

I am working with this Pandas DataFrame in Python. File heat Farheit Temp_Rating 1 YesQ 75 N/A 1 NoR 115 N/A 1 YesA 63 N/A 1 NoT 83 41 1 NoY 100 80 1 YesZ 56 12 2 YesQ 111 N/A 2 NoR 60 N/A 2 YesA 19 N/A 2 NoT 106 77 2 NoY 45 21 2 YesZ 40 54 3 YesQ 84 N/A 3 NoR 67 N/A 3 YesA 94 N/A 3 NoT 68 39 3 NoY 63 46 3 YesZ 34 81 I need to replace all NaNs in the Temp_Rating column with the value from the Farheit column. This is what I need: File heat Temp_Rating 1 YesQ 75 1 NoR 115 1 YesA 63 1 YesQ 41 1 NoR 80 1 YesA 12 2 YesQ 111 2 NoR 60 2 YesA 19 2 NoT 77 2 NoY 21 2 YesZ 54 3 YesQ 84 3 NoR 67 3 YesA 94 3 NoT 39 3 NoY 46 3 YesZ 81 If I do a Boolean selection, I can pick out only one of these columns at a time. The problem is if I then try to join them, I am not able to do this while preserving the correct order. How can I only find Temp_Rating rows with the NaNs and replace them with the value in the same row of the Farheit column?

Output: Assuming your DataFrame is in df: df.Temp_Rating.fillna(df.Farheit, inplace=True) del df['Farheit'] df.columns = 'File heat Observations'.split() First replace any NaN values with the corresponding value of df.Farheit. Delete the 'Farheit' column. Then rename the columns. Here's the resulting DataFrame: File heat Observations 0 1 YesQ 75 1 1 NoR 115 2 1 YesA 63 3 1 NoT 41 4 1 NoY 80 5 1 YesZ 12 6 2 YesQ 111 7 2 NoR 60 8 2 YesA 19 9 2 NoT 77 10 2 NoY 21 11 2 YesZ 54 12 3 YesQ 84 13 3 NoR 67 14 3 YesA 94 15 3 NoT 39 16 3 NoY 46 17 3 YesZ 81


Instruction:
Python check if variable isinstance of any type in list

How do I compactly perform the following: var = 7.0 var_is_good = ( isinstance(var, classinfo1) or isinstance(var, classinfo2) or isinstance(var, classinfo3) )

Output: isinstance() takes a tuple of classes for the second argument. It'll return true if the first argument is an instance of any of the types in that sequence: isinstance(var, (classinfo1, classinfo2, classinfo3)) In other words, isinstance() already offers this functionality, out of the box. From the isinstance() documentation: If classinfo is neither a class object nor a type object, it may be a tuple of class or type objects, or may recursively contain other such tuples (other sequence types are not accepted). Emphasis mine; note the recursive nature; (classinfo1, (classinfo2, classinfo3)) is also a valid option.