Instruction:
Tensorflow - ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type float)

Continuation from previous question: Tensorflow - TypeError: 'int' object is not iterable My training data is a list of lists each comprised of 1000 floats. For example, x_train[0] = [0.0, 0.0, 0.1, 0.25, 0.5, ...] Here is my model: model = Sequential() model.add(LSTM(128, activation='relu', input_shape=(1000, 1), return_sequences=True)) model.add(Dropout(0.2)) model.add(LSTM(128, activation='relu')) model.add(Dropout(0.2)) model.add(Dense(32, activation='relu')) model.add(Dropout(0.2)) model.add(Dense(1, activation='sigmoid')) opt = tf.keras.optimizers.Adam(lr=1e-3, decay=1e-5) model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy']) model.fit(x_train, y_train, epochs=3, validation_data=(x_test, y_test)) Here is the error I'm getting: Traceback (most recent call last): File "C:\Users\bencu\Desktop\ProjectFiles\Code\Program.py", line 88, in FitModel model.fit(x_train, y_train, epochs=3, validation_data=(x_test, y_test)) File "C:\Users\bencu\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\training.py", line 728, in fit use_multiprocessing=use_multiprocessing) File "C:\Users\bencu\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py", line 224, in fit distribution_strategy=strategy) File "C:\Users\bencu\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py", line 547, in _process_training_inputs use_multiprocessing=use_multiprocessing) File "C:\Users\bencu\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py", line 606, in _process_inputs use_multiprocessing=use_multiprocessing) File "C:\Users\bencu\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\data_adapter.py", line 479, in __init__ batch_size=batch_size, shuffle=shuffle, **kwargs) File "C:\Users\bencu\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\data_adapter.py", line 321, in __init__ dataset_ops.DatasetV2.from_tensors(inputs).repeat() File "C:\Users\bencu\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\data\ops\dataset_ops.py", line 414, in from_tensors return TensorDataset(tensors) File "C:\Users\bencu\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\data\ops\dataset_ops.py", line 2335, in __init__ element = structure.normalize_element(element) File "C:\Users\bencu\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\data\util\structure.py", line 111, in normalize_element ops.convert_to_tensor(t, name="component_%d" % i)) File "C:\Users\bencu\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1184, in convert_to_tensor return convert_to_tensor_v2(value, dtype, preferred_dtype, name) File "C:\Users\bencu\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1242, in convert_to_tensor_v2 as_ref=False) File "C:\Users\bencu\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1296, in internal_convert_to_tensor ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref) File "C:\Users\bencu\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\framework\tensor_conversion_registry.py", line 52, in _default_conversion_function return constant_op.constant(value, dtype, name=name) File "C:\Users\bencu\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\framework\constant_op.py", line 227, in constant allow_broadcast=True) File "C:\Users\bencu\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\framework\constant_op.py", line 235, in _constant_impl t = convert_to_eager_tensor(value, ctx, dtype) File "C:\Users\bencu\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\framework\constant_op.py", line 96, in convert_to_eager_tensor return ops.EagerTensor(value, ctx.device_name, dtype) ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type float). I've tried googling the error myself, I found something about using the tf.convert_to_tensor function. I tried passing my training and testing lists through this but the function won't take them.

Output: TL;DR Several possible errors, most fixed with x = np.asarray(x).astype('float32'). Others may be faulty data preprocessing; ensure everything is properly formatted (categoricals, nans, strings, etc). Below shows what the model expects: [print(i.shape, i.dtype) for i in model.inputs] [print(o.shape, o.dtype) for o in model.outputs] [print(l.name, l.input_shape, l.dtype) for l in model.layers] The problem's rooted in using lists as inputs, as opposed to Numpy arrays; Keras/TF doesn't support former. A simple conversion is: x_array = np.asarray(x_list). The next step's to ensure data is fed in expected format; for LSTM, that'd be a 3D tensor with dimensions (batch_size, timesteps, features) - or equivalently, (num_samples, timesteps, channels). Lastly, as a debug pro-tip, print ALL the shapes for your data. Code accomplishing all of the above, below: Sequences = np.asarray(Sequences) Targets = np.asarray(Targets) show_shapes() Sequences = np.expand_dims(Sequences, -1) Targets = np.expand_dims(Targets, -1) show_shapes() # OUTPUTS Expected: (num_samples, timesteps, channels) Sequences: (200, 1000) Targets: (200,) Expected: (num_samples, timesteps, channels) Sequences: (200, 1000, 1) Targets: (200, 1) As a bonus tip, I notice you're running via main(), so your IDE probably lacks a Jupyter-like cell-based execution; I strongly recommend the Spyder IDE. It's as simple as adding # In[], and pressing Ctrl + Enter below: Function used: def show_shapes(): # can make yours to take inputs; this'll use local variable values print("Expected: (num_samples, timesteps, channels)") print("Sequences: {}".format(Sequences.shape)) print("Targets: {}".format(Targets.shape))


Instruction:
What's the difference between MySQLdb, mysqlclient and MySQL connector/Python?

So I've been trying to do some database update with python and while setting up the whole dev environment, I came across these three things which made me dizzy. There's MySQLdb There's mysqlclient And then there's a mysql connector python What's each of them, the difference and where to use them? Thanks

Output: MySQLdb is a thin python wrapper around C module which implements API for MySQL database. There was MySQLDb1 version of wrapper used some time ago and now it is considered to be a legacy. As MySQLDb1 started evolving to MySQLDb2 with bug fixes and Python3 support, a MySQLDb1 was forked and here is how mysqlclient appeared, with bugfixes and Python3 support. Sum up, so now we have MySQLDb2 which is not ready for production use, MySQLDb1 as an outdated driver and a community supported mysqlclient with bug fixes and Python3 support. Now, to solve that mess, MySQL provides their own version of MySQL adapter - mysql connector, an all-in python module that uses MySQL API with no C modules dependencies and only standard python modules used. So now the question comes down to: mysqlclient vs mysql connector. As for me, I would go with officially supported library, however mysqlclient should be a good choice as well. Both of them are being actively updated with fixes and new features which you can see by active commits in last days. Note: I did not have much experience with them, so there might be cases when one or another does not suite your needs. Both libraries follow PEP-249 standard which means you should be fine with at least base functionality everywhere. Installation and Dependencies mysqlclient As a fork of C wrapper it requires C modules to work with MySQL which adds python header files to build these extensions (read python-dev). Installation depends on the system you use, just make sure you aware of package names and can install them. mysql connector Main documentation is pretty clear, however you should be aware of Protobuf C++ dependency (for mysql connector versions >= 2.2.3 < 2.2.6).


Instruction:
Why does using "from __future__ import print_function" break Python2-style print?

I tried this code in Python 2.7: from __future__ import print_function import sys, os, time for x in range(0,10): print x, sep=' ', end='' time.sleep(1) But I get an error that says: $ python2 xy.py File "xy.py", line 5 print x, sep=' ', end='' ^ SyntaxError: invalid syntax $ I thought that using the __future__ import should make it possible to use sep and end in a print statement, but now it apparently doesn't work at all. Why not?

Output: The whole point of from __future__ import print_function is to bring the print function from Python 3 into Python 2.6+. Thus, it must be used like a function here: from __future__ import print_function import sys, os, time for x in range(0,10): print(x, sep=' ', end='') # No need for sep here, but okay :) time.sleep(1) __future__ statements change fundamental things about the language. From the documentation: A future statement is recognized and treated specially at compile time: Changes to the semantics of core constructs are often implemented by generating different code. It may even be the case that a new feature introduces new incompatible syntax (such as a new reserved word), in which case the compiler may need to parse the module differently. Such decisions cannot be pushed off until runtime. (For the same reason, they must also appear first in the source code, before any other imports. The only things that can precede a __future__ statement are the module docstring, comments, blank lines, and other future statements.)


Instruction:
Making an asynchronous task in Flask

I am writing an application in Flask, which works really well except that WSGI is synchronous and blocking. I have one task in particular which calls out to a third party API and that task can take several minutes to complete. I would like to make that call (it's actually a series of calls) and let it run. while control is returned to Flask. My view looks like: @app.route('/render/<id>', methods=['POST']) def render_script(id=None): ... data = json.loads(request.data) text_list = data.get('text_list') final_file = audio_class.render_audio(data=text_list) # do stuff return Response( mimetype='application/json', status=200 ) Now, what I want to do is have the line final_file = audio_class.render_audio() run and provide a callback to be executed when the method returns, whilst Flask can continue to process requests. This is the only task which I need Flask to run asynchronously, and I would like some advice on how best to implement this. I have looked at Twisted and Klein, but I'm not sure they are overkill, as maybe Threading would suffice. Or maybe Celery is a good choice for this?

Output: I would use Celery to handle the asynchronous task for you. You'll need to install a broker to serve as your task queue (RabbitMQ and Redis are recommended). app.py: from flask import Flask from celery import Celery broker_url = 'amqp://guest@localhost' # Broker URL for RabbitMQ task queue app = Flask(__name__) celery = Celery(app.name, broker=broker_url) celery.config_from_object('celeryconfig') # Your celery configurations in a celeryconfig.py @celery.task(bind=True) def some_long_task(self, x, y): # Do some long task ... @app.route('/render/<id>', methods=['POST']) def render_script(id=None): ... data = json.loads(request.data) text_list = data.get('text_list') final_file = audio_class.render_audio(data=text_list) some_long_task.delay(x, y) # Call your async task and pass whatever necessary variables return Response( mimetype='application/json', status=200 ) Run your Flask app, and start another process to run your celery worker. $ celery worker -A app.celery --loglevel=debug I would also refer to Miguel Gringberg's write up for a more in depth guide to using Celery with Flask.


Instruction:
Apply function to each cell in DataFrame

I have a dataframe that may look like this: A B C foo bar foo bar bar foo foo bar I want to look through every element of each row (or every element of each column) and apply the following function to get the subsequent dataframe: def foo_bar(x): return x.replace('foo', 'wow') After applying the function, my dataframe will look like this: A B C wow bar wow bar bar wow wow bar Is there a simple one-liner that can apply a function to each cell? This is a simplistic example so there may be an easier way to execute this specific example other than applying a function, but what I am really asking about is how to apply a function in every cell within a dataframe.

Output: You can use map() which is concise for your case. df.map(foo_bar) # A B C #0 wow bar wow bar #1 bar wow wow bar Another option is to vectorize your function and then use apply method: import numpy as np df.apply(np.vectorize(foo_bar)) # A B C #0 wow bar wow bar #1 bar wow wow bar


Instruction:
Times-two faster than bit-shift, for Python 3.x integers?

I was looking at the source of sorted_containers and was surprised to see this line: self._load, self._twice, self._half = load, load * 2, load >> 1 Here load is an integer. Why use bit shift in one place, and multiplication in another? It seems reasonable that bit shifting may be faster than integral division by 2, but why not replace the multiplication by a shift as well? I benchmarked the the following cases: (times, divide) (shift, shift) (times, shift) (shift, divide) and found that #3 is consistently faster than other alternatives: # self._load, self._twice, self._half = load, load * 2, load >> 1 import random import timeit import pandas as pd x = random.randint(10 ** 3, 10 ** 6) def test_naive(): a, b, c = x, 2 * x, x // 2 def test_shift(): a, b, c = x, x << 1, x >> 1 def test_mixed(): a, b, c = x, x * 2, x >> 1 def test_mixed_swapped(): a, b, c = x, x << 1, x // 2 def observe(k): print(k) return { 'naive': timeit.timeit(test_naive), 'shift': timeit.timeit(test_shift), 'mixed': timeit.timeit(test_mixed), 'mixed_swapped': timeit.timeit(test_mixed_swapped), } def get_observations(): return pd.DataFrame([observe(k) for k in range(100)]) The question: Is my test valid? If so, why is (multiply, shift) faster than (shift, shift)? I run Python 3.5 on Ubuntu 14.04. Edit Above is the original statement of the question. Dan Getz provides an excellent explanation in his answer. For the sake of completeness, here are sample illustrations for larger x when multiplication optimizations do not apply.

Output: This seems to be because multiplication of small numbers is optimized in CPython 3.5, in a way that left shifts by small numbers are not. Positive left shifts always create a larger integer object to store the result, as part of the calculation, while for multiplications of the sort you used in your test, a special optimization avoids this and creates an integer object of the correct size. This can be seen in the source code of Python's integer implementation. Because integers in Python are arbitrary-precision, they are stored as arrays of integer "digits", with a limit on the number of bits per integer digit. So in the general case, operations involving integers are not single operations, but instead need to handle the case of multiple "digits". In pyport.h, this bit limit is defined as 30 bits on 64-bit platform, or 15 bits otherwise. (I'll just call this 30 from here on to keep the explanation simple. But note that if you were using Python compiled for 32-bit, your benchmark's result would depend on if x were less than 32,768 or not.) When an operation's inputs and outputs stay within this 30-bit limit, the operation can be handled in an optimized way instead of the general way. The beginning of the integer multiplication implementation is as follows: static PyObject * long_mul(PyLongObject *a, PyLongObject *b) { PyLongObject *z; CHECK_BINOP(a, b); /* fast path for single-digit multiplication */ if (Py_ABS(Py_SIZE(a)) <= 1 && Py_ABS(Py_SIZE(b)) <= 1) { stwodigits v = (stwodigits)(MEDIUM_VALUE(a)) * MEDIUM_VALUE(b); #ifdef HAVE_LONG_LONG return PyLong_FromLongLong((PY_LONG_LONG)v); #else /* if we don't have long long then we're almost certainly using 15-bit digits, so v will fit in a long. In the unlikely event that we're using 30-bit digits on a platform without long long, a large v will just cause us to fall through to the general multiplication code below. */ if (v >= LONG_MIN && v <= LONG_MAX) return PyLong_FromLong((long)v); #endif } So when multiplying two integers where each fits in a 30-bit digit, this is done as a direct multiplication by the CPython interpreter, instead of working with the integers as arrays. (MEDIUM_VALUE() called on a positive integer object simply gets its first 30-bit digit.) If the result fits in a single 30-bit digit, PyLong_FromLongLong() will notice this in a relatively small number of operations, and create a single-digit integer object to store it. In contrast, left shifts are not optimized this way, and every left shift deals with the integer being shifted as an array. In particular, if you look at the source code for long_lshift(), in the case of a small but positive left shift, a 2-digit integer object is always created, if only to have its length truncated to 1 later: (my comments in /*** ***/) static PyObject * long_lshift(PyObject *v, PyObject *w) { /*** ... ***/ wordshift = shiftby / PyLong_SHIFT; /*** zero for small w ***/ remshift = shiftby - wordshift * PyLong_SHIFT; /*** w for small w ***/ oldsize = Py_ABS(Py_SIZE(a)); /*** 1 for small v > 0 ***/ newsize = oldsize + wordshift; if (remshift) ++newsize; /*** here newsize becomes at least 2 for w > 0, v > 0 ***/ z = _PyLong_New(newsize); /*** ... ***/ } Integer division You didn't ask about the worse performance of integer floor division compared to right shifts, because that fit your (and my) expectations. But dividing a small positive number by another small positive number is not as optimized as small multiplications, either. Every // computes both the quotient and the remainder using the function long_divrem(). This remainder is computed for a small divisor with a multiplication, and is stored in a newly-allocated integer object, which in this situation is immediately discarded. Or at least, that was the case when this question was originally asked. In CPython 3.6, a fast path for small int // was added, so // now beats >> for small ints too.


Instruction:
What's the difference between "pip install" and "python -m pip install"?

I have a local version of Python 3.4.1 and I can run python -m pip install, but I'm unable to find the pip binary to run pip install. What's the difference between these two?

Output: They do exactly the same thing, assuming pip is using the same version of Python as the python executable. The docs for distributing Python modules were just updated to suggest using python -m pip instead of the pip executable, because it allows you to be explicit about which version of Python to use. In systems with more than one version of Python installed, it's not always clear which one pip is linked to. Here's some more concrete "proof" that both commands should do the same thing, beyond just trusting my word and the bug report I linked :) If you take a look at the pip executable script, it's just doing this: from pkg_resources import load_entry_point <snip> load_entry_point('pip==1.5.4', 'console_scripts', 'pip')() It's calling load_entry_point, which returns a function, and then executing that function. The entry point it's using is called 'console_scripts'. If you look at the entry_points.txt file for pip (/usr/lib/python2.7/dist-packages/pip-1.5.4.egg-info/entry_points.txt on my Ubuntu machine), you'll see this: [console_scripts] pip = pip:main pip2.7 = pip:main pip2 = pip:main So the entry point returned is the main function in the pip module. When you run python -m pip, you're executing the __main__.py script inside the pip package. That looks like this: import sys from .runner import run if __name__ == '__main__': exit = run() if exit: sys.exit(exit) And the runner.run function looks like this: def run(): base = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) ## FIXME: this is kind of crude; if we could create a fake pip ## module, then exec into it and update pip.__path__ properly, we ## wouldn't have to update sys.path: sys.path.insert(0, base) import pip return pip.main() As you can see, it's just calling the pip.main function, too. So both commands end up calling the same main function in pip/__init__.py.


Instruction:
Saving interactive Matplotlib figures

Is there a way to save a Matplotlib figure such that it can be re-opened and have typical interaction restored? (Like the .fig format in MATLAB?) I find myself running the same scripts many times to generate these interactive figures. Or I'm sending my colleagues multiple static PNG files to show different aspects of a plot. I'd rather send the figure object and have them interact with it themselves.

Output: Matplotlib since version 1.2 allowed you to pickle figures. As the release notes state, it is an experimental feature and does not support saving a figure in one matplotlib version and opening in another. It's also generally unsecure to restore a pickle from an untrusted source. For sharing/later editing plots (that require significant data processing first and may need to be tweaked months later say during peer review for a scientific publication), I still recommend the workflow of (1) have a data processing script that before generating a plot saves the processed data (that goes into your plot) into a file, and (2) have a separate plot generation script (that you adjust as necessary) to recreate the plot. This way for each plot you can quickly run a script and re-generate it (and quickly copy over your plot settings with new data). That said, pickling a figure could be convenient for short term/interactive/exploratory data analysis. If you are using a really old version that doesn't have support yet, I'd suggest either (a) separate processing the data from generating the figure (which saves data with a unique name) and write a figure generating script (loading a specified file of the saved data) and editing as you see fit or (b) save as PDF/SVG/PostScript format and edit in some fancy figure editor like Adobe Illustrator (or Inkscape).


Instruction:
What is an 'endpoint' in Flask?

The Flask documentation shows: add_url_rule(*args, **kwargs) Connects a URL rule. Works exactly like the route() decorator. If a view_func is provided it will be registered with the endpoint. endpoint – the endpoint for the registered URL rule. Flask itself assumes the name of the view function as endpoint What exactly is meant by an "endpoint"?

Output: How Flask Routing Works The entire idea of Flask (and the underlying Werkzeug library) is to map URL paths to some logic that you will run (typically, the "view function"). Your basic view is defined like this: @app.route('/greeting/<name>') def give_greeting(name): return 'Hello, {0}!'.format(name) Note that the function you referred to (add_url_rule) achieves the same goal, just without using the decorator notation. Therefore, the following is the same: # No "route" decorator here. We will add routing using a different method below. def give_greeting(name): return 'Hello, {0}!'.format(name) app.add_url_rule('/greeting/<name>', 'give_greeting', give_greeting) Let's say your website is located at 'www.example.org' and uses the above view. The user enters the following URL into their browser: http://www.example.org/greeting/Mark The job of Flask is to take this URL, figure out what the user wants to do, and pass it on to one of your many python functions for handling. It takes the path: /greeting/Mark ...and matches it to the list of routes. In our case, we defined this path to go to the give_greeting function. However, while this is the typical way that you might go about creating a view, it actually abstracts some extra info from you. Behind the scenes, Flask did not make the leap directly from URL to the view function that should handle this request. It does not simply say... URL (http://www.example.org/greeting/Mark) should be handled by View Function (the function "give_greeting") Actually, it there is another step, where it maps the URL to an endpoint: URL (http://www.example.org/greeting/Mark) should be handled by Endpoint "give_greeting". Requests to Endpoint "give_greeting" should be handled by View Function "give_greeting" Basically, the "endpoint" is an identifier that is used in determining what logical unit of your code should handle the request. Normally, an endpoint is just the name of a view function. However, you can actually change the endpoint, as is done in the following example. @app.route('/greeting/<name>', endpoint='say_hello') def give_greeting(name): return 'Hello, {0}!'.format(name) Now, when Flask routes the request, the logic looks like this: URL (http://www.example.org/greeting/Mark) should be handled by Endpoint "say_hello". Endpoint "say_hello" should be handled by View Function "give_greeting" How You Use the Endpoint The endpoint is commonly used for the "reverse lookup". For example, in one view of your Flask application, you want to reference another view (perhaps when you are linking from one area of the site to another). Rather than hard-code the URL, you can use url_for(). Assume the following @app.route('/') def index(): print url_for('give_greeting', name='Mark') # This will print '/greeting/Mark' @app.route('/greeting/<name>') def give_greeting(name): return 'Hello, {0}!'.format(name) This is advantageous, as now we can change the URLs of our application without needing to change the line where we reference that resource. Why not just always use the name of the view function? One question that might come up is the following: "Why do we need this extra layer?" Why map a path to an endpoint, then an endpoint to a view function? Why not just skip that middle step? The reason is because it is more powerful this way. For example, Flask Blueprints allow you to split your application into various parts. I might have all of my admin-side resources in a blueprint called "admin", and all of my user-level resources in an endpoint called "user". Blueprints allow you to separate these into namespaces. For example... main.py: from flask import Flask, Blueprint from admin import admin from user import user app = Flask(__name__) app.register_blueprint(admin, url_prefix='/admin') app.register_blueprint(user, url_prefix='/user') admin.py: admin = Blueprint('admin', __name__) @admin.route('/greeting') def greeting(): return 'Hello, administrative user!' user.py: user = Blueprint('user', __name__) @user.route('/greeting') def greeting(): return 'Hello, lowly normal user!' Note that in both blueprints, the '/greeting' route is a function called "greeting". If I wanted to refer to the admin "greeting" function, I couldn't just say "greeting" because there is also a user "greeting" function. Endpoints allow for a sort of namespacing by having you specify the name of the blueprint as part of the endpoint. So, I could do the following... print url_for('admin.greeting') # Prints '/admin/greeting' print url_for('user.greeting') # Prints '/user/greeting'


Instruction:
Logistic regression python solvers' definitions

I am using the logistic regression function from sklearn, and was wondering what each of the solver is actually doing behind the scenes to solve the optimization problem. Can someone briefly describe what "newton-cg", "sag", "lbfgs" and "liblinear" are doing?

Output: Well, I hope I'm not too late for the party! Let me first try to establish some intuition before digging into loads of information (warning: this is not a brief comparison, TL;DR) Introduction A hypothesis h(x), takes an input and gives us the estimated output value. This hypothesis can be as simple as a one-variable linear equation, .. up to a very complicated and long multivariate equation with respect to the type of algorithm we’re using (e.g. linear regression, logistic regression..etc). Our task is to find the best Parameters (a.k.a Thetas or Weights) that give us the least error in predicting the output. We call the function that calculates this error a Cost or Loss Function, and apparently, our goal is to minimize the error in order to get the best-predicted output! One more thing to recall is, the relation between the parameter value and its effect on the cost function (i.e. the error) looks like a bell curve (i.e. Quadratic; recall this because it’s important). So if we start at any point in that curve and keep taking the derivative (i.e. tangent line) of each point we stop at (assuming it's a univariate problem, otherwise, if we have multiple features, we take the partial derivative), we will end up at what so-called the Global Optima as shown in this image: If we take the partial derivative at the minimum cost point (i.e. global optima) we find the slope of the tangent line = 0 (then we know that we reached our target). That’s valid only if we have a Convex Cost Function, but if we don’t, we may end up stuck at what is called Local Optima; consider this non-convex function: Now you should have the intuition about the heck relationship between what we are doing and the terms: Derivative, Tangent Line, Cost Function, Hypothesis ..etc. Side Note: The above-mentioned intuition is also related to the Gradient Descent Algorithm (see later). Background Linear Approximation: Given a function, f(x), we can find its tangent at x=a. The equation of the tangent line L(x) is: L(x)=f(a)+f′(a)(x−a). Take a look at the following graph of a function and its tangent line: From this graph we can see that near x=a, the tangent line and the function have nearly the same graph. On occasion, we will use the tangent line, L(x), as an approximation to the function, f(x), near x=a. In these cases, we call the tangent line the "Linear Approximation" to the function at x=a. Quadratic Approximation: Same as a linear approximation, yet this time we are dealing with a curve where we cannot find the point near to 0 by using only the tangent line. Instead, we use the parabola as it's shown in the following graph: In order to fit a good parabola, both parabola and quadratic function should have the same value, the same first derivative, AND the same second derivative. The formula will be (just out of curiosity): Qa(x) = f(a) + f'(a)(x-a) + f''(a)(x-a)2/2 Now we should be ready to do the comparison in detail. Comparison between the methods 1. Newton’s Method Recall the motivation for the gradient descent step at x: we minimize the quadratic function (i.e. Cost Function). Newton’s method uses in a sense a better quadratic function minimisation. It's better because it uses the quadratic approximation (i.e. first AND second partial derivatives). You can imagine it as a twisted Gradient Descent with the Hessian (the Hessian is a square matrix of second-order partial derivatives of order n X n). Moreover, the geometric interpretation of Newton's method is that at each iteration one approximates f(x) by a quadratic function around xn, and then takes a step towards the maximum/minimum of that quadratic function (in higher dimensions, this may also be a saddle point). Note that if f(x) happens to be a quadratic function, then the exact extremum is found in one step. Drawbacks: It’s computationally expensive because of the Hessian Matrix (i.e. second partial derivatives calculations). It attracts to Saddle Points which are common in multivariable optimization (i.e. a point that its partial derivatives disagree over whether this input should be a maximum or a minimum point!). 2. Limited-memory Broyden–Fletcher–Goldfarb–Shanno Algorithm: In a nutshell, it is an analogue of Newton’s Method, yet here the Hessian matrix is approximated using updates specified by gradient evaluations (or approximate gradient evaluations). In other words, using estimation to the inverse Hessian matrix. The term Limited-memory simply means it stores only a few vectors that represent the approximation implicitly. If I dare say that when the dataset is small, L-BFGS relatively performs the best compared to other methods especially because it saves a lot of memory, however, there are some “serious” drawbacks such that if it is unsafeguarded, it may not converge to anything. Side note: This solver has become the default solver in sklearn LogisticRegression since version 0.22, replacing LIBLINEAR. 3. A Library for Large Linear Classification: It’s a linear classification that supports logistic regression and linear support vector machines. The solver uses a Coordinate Descent (CD) algorithm that solves optimization problems by successively performing approximate minimization along coordinate directions or coordinate hyperplanes. LIBLINEAR is the winner of the ICML 2008 large-scale learning challenge. It applies automatic parameter selection (a.k.a L1 Regularization) and it’s recommended when you have high dimension dataset (recommended for solving large-scale classification problems) Drawbacks: It may get stuck at a non-stationary point (i.e. non-optima) if the level curves of a function are not smooth. Also cannot run in parallel. It cannot learn a true multinomial (multiclass) model; instead, the optimization problem is decomposed in a “one-vs-rest” fashion, so separate binary classifiers are trained for all classes. Side note: According to Scikit Documentation: The “liblinear” solver was the one used by default for historical reasons before version 0.22. Since then, the default use is Limited-memory Broyden–Fletcher–Goldfarb–Shanno Algorithm. 4. Stochastic Average Gradient: The SAG method optimizes the sum of a finite number of smooth convex functions. Like stochastic gradient (SG) methods, the SAG method's iteration cost is independent of the number of terms in the sum. However, by incorporating a memory of previous gradient values, the SAG method achieves a faster convergence rate than black-box SG methods. It is faster than other solvers for large datasets when both the number of samples and the number of features are large. Drawbacks: It only supports L2 penalization. This is not really a drawback, but more like a comparison: although SAG is suitable for large datasets, with a memory cost of O(N), it can be less practical for very large N (as the most recent gradient evaluation for each function needs to be maintained in the memory). This is usually not a problem, but a better option would be SVRG 1, 2 which is unfortunately not implemented in scikit-learn! 5. SAGA: The SAGA solver is a variant of SAG that also supports the non-smooth penalty L1 option (i.e. L1 Regularization). This is therefore the solver of choice for sparse multinomial logistic regression. It also has a better theoretical convergence compared to SAG. Drawbacks: This is not really a drawback, but more like a comparison: SAGA is similar to SAG with regard to memory cost. That's it's suitable for large datasets, yet in edge cases where the dataset is very large, the SVRG 1, 2 would be a better option (unfortunately not implemented in scikit-learn)! Side note: According to Scikit Documentation: The SAGA solver is often the best choice. Please note the attributes "Large" and "Small" used in Scikit-Learn and in this comparison are relative. AFAIK, there is no universal unanimous and accurate definition of the dataset boundaries to be considered as "Large", "Too Large", "Small", "Too Small"...etc! Summary The following table is taken from Scikit Documentation Updated Table from the same link above (accessed 02/11/2021):


Instruction:
How to exit an if clause

What sorts of methods exist for prematurely exiting an if clause? There are times when I'm writing code and want to put a break statement inside of an if clause, only to remember that those can only be used for loops. Lets take the following code as an example: if some_condition: ... if condition_a: # do something # and then exit the outer if block ... if condition_b: # do something # and then exit the outer if block # more code here I can think of one way to do this: assuming the exit cases happen within nested if statements, wrap the remaining code in a big else block. Example: if some_condition: ... if condition_a: # do something # and then exit the outer if block else: ... if condition_b: # do something # and then exit the outer if block else: # more code here The problem with this is that more exit locations mean more nesting/indented code. Alternatively, I could write my code to have the if clauses be as small as possible and not require any exits. Does anyone know of a good/better way to exit an if clause? If there are any associated else-if and else clauses, I figure that exiting would skip over them.

Output: This method works for ifs, multiple nested loops, and other constructs that you can't break from easily. Wrap the code in its own function. Instead of break, use return. Example: def some_function(): if condition_a: # do something and return early ... return ... if condition_b: # do something else and return early ... return ... return if outer_condition: ... some_function() ...


Instruction:
Using tqdm progress bar in a while loop

I am making a code that simulates a pawn going around a monopoly board a million times. I would like to have a tqdm progress bar that is updated every time a turn around the board is achieved. Below is my current code. I am using a while loop which stops when the number of turns around the board surpasses the desired number. import os from openpyxl import Workbook from monopolyfct import * def main(runs, fileOutput): ### EXCEL SETUP ### theWorkbook = Workbook() # Creates the workbook interface. defaultSheet = theWorkbook.active # Creates the used worksheet. currentData = ["Current Table Turn", "Current Tile"] # Makes EXCEL column titles. defaultSheet.append(currentData) # Appends column titles. ### CONTENT SETUP ### currentData = [1, 0] # Sets starting position. defaultSheet.append(currentData) # Appends starting position. while currentData[0] <= runs: ### ROLLING THE DICES PROCESS ### dices = twinDiceRoll() currentData[1] += dices[2] # Updating the current tile ### SURPASSING THE NUMBER OF TILES ONBOARD ### if currentData[1] > 37: # If more than a table turn is achieved, currentData[0] += 1 # One more turn is registered currentData[1] -= 38 # Update the tile to one coresponding to a board tile. else: pass ### APPENDING AQUIRED DATA ### defaultSheet.append(currentData) ### MANAGIING SPECIAL TILES ### if currentData[1] == 2 or 15 or 31: # Community chess pass #TODO: Make a mechanic simulating the community chest card draw and it's related action. elif currentData[1] == 5 or 20 or 34: # Chance pass #TODO: Make a mechanic simulating the chance card draw and it's related action. elif currentData[1] == 28: # Go to Jail pass #TODO: Make a mechanic simulating the entire jail process ### TWIN DICE ROLL EXCEPTION ### if dices[3] is True: # If the dices roll a double, pass #TODO: Make a mechanic considering that three doubles sends one to Jail. ### STORING THE ACCUMULATED DATA ### theWorkbook.save(fileOutput) # Compiles the data in a .xlxs file. if __name__ == "__main__": terminalWidth = os.get_terminal_size().columns # Gets current terminal width. space(3) print("Python Monopoly Statistics Renderer".upper().center(terminalWidth)) # Prints the title. print("(PMSR)".center(terminalWidth)) # Prints the acronym. space(2) runs = int(request("For how many table turns do you want the simulation to run?")) # Prompts for the desired run ammount #runs = 1000 fileOutput = request("What should be the name of the file in which statistics are stored?") # Prompts for the desired store filename #fileOutput = "test" fileOutput += ".xlsx" # Adds file extension to filename main(runs, fileOutput)

Output: You can use manual control in tqdm by specifying a total argument in the constructor. Verbatim from the manual: with tqdm(total=100) as pbar: for i in range(10): sleep(0.1) pbar.update(10) UPDATE To manually control the tqdm without the context manager (aka with statement), you will need to close the progress bar after you are done using it. Here is another example from the manual: pbar = tqdm(total=100) for i in range(10): sleep(0.1) pbar.update(10) pbar.close() For this to work you need to know the total number of expected runs. In your code it could look something like ... pbar = tqdm(total = runs+1) while currentData[0] <= runs: ### ROLLING THE DICES PROCESS ### dices = twinDiceRoll() currentData[1] += dices[2] # Updating the current tile ### SURPASSING THE NUMBER OF TILES ONBOARD ### if currentData[1] > 37: # If more than a table turn is achieved, currentData[0] += 1 # One more turn is registered currentData[1] -= 38 # Update the tile to one coresponding to a board tile. pbar.update(1) else: pass ... pbar.close() However, this code isn't perfect: consider if the currentData[1] is always less than 37 -- the progress bar will just stop and not update. If you try to update it in the else:... part, you might violate the total upper bound. This is a start tho :)


Instruction:
Adding meta-information/metadata to pandas DataFrame

Is it possible to add some meta-information/metadata to a pandas DataFrame? For example, the instrument's name used to measure the data, the instrument responsible, etc. One workaround would be to create a column with that information, but it seems wasteful to store a single piece of information in every row!

Output: Sure, like most Python objects, you can attach new attributes to a pandas.DataFrame: import pandas as pd df = pd.DataFrame([]) df.instrument_name = 'Binky' Note, however, that while you can attach attributes to a DataFrame, operations performed on the DataFrame (such as groupby, pivot, join, assign or loc to name just a few) may return a new DataFrame without the metadata attached. Pandas does not yet have a robust method of propagating metadata attached to DataFrames. Preserving the metadata in a file is possible. You can find an example of how to store metadata in an HDF5 file here.


Instruction:
How to completely uninstall a Django app?

What is the procedure for completely uninstalling a Django app, complete with database removal?

Output: Django < 1.7 has a handy management command that will give you the necessary SQL to drop all the tables for an app. See the sqlclear docs for more information. Basically, running ./manage.py sqlclear my_app_name gets you get the SQL statements that should be executed to get rid of all traces of the app in your DB. You still need to copy and paste (or pipe) those statements into your SQL client. For Django 1.7 and up, use ./manage.py migrate my_app_name zero (see the migrate docs), which runs the database cleaning automatically. To remove the app from your project, all you need to do is remove it from INSTALLED_APPS in your project's settings.py. Django will no longer load the app. If you no longer want the app's files hanging around, delete the app directory from your project directory or other location on your PYTHONPATH where it resides. (optional) If the app stored media files, cache files, or other temporary files somewhere, you may want to delete those as well. Also be wary of lingering session data that might be leftover from the app. (optional) I would also remove any stale content types. Like so. from django.contrib.contenttypes.models import ContentType for c in ContentType.objects.all(): if not c.model_class(): print "deleting %s"%c # print(f"deleting {c}") # for Python 3.6+ c.delete()


Instruction:
Are for-loops in pandas really bad? When should I care?

Are for loops really "bad"? If not, in what situation(s) would they be better than using a more conventional "vectorized" approach?1 I am familiar with the concept of "vectorization", and how pandas employs vectorized techniques to speed up computation. Vectorized functions broadcast operations over the entire series or DataFrame to achieve speedups much greater than conventionally iterating over the data. However, I am quite surprised to see a lot of code (including from answers on Stack Overflow) offering solutions to problems that involve looping through data using for loops and list comprehensions. The documentation and API say that loops are "bad", and that one should "never" iterate over arrays, series, or DataFrames. So, how come I sometimes see users suggesting loop-based solutions? 1 - While it is true that the question sounds somewhat broad, the truth is that there are very specific situations when for loops are usually better than conventionally iterating over data. This post aims to capture this for posterity.

Output: TLDR; No, for loops are not blanket "bad", at least, not always. It is probably more accurate to say that some vectorized operations are slower than iterating, versus saying that iteration is faster than some vectorized operations. Knowing when and why is key to getting the most performance out of your code. In a nutshell, these are the situations where it is worth considering an alternative to vectorized pandas functions: When your data is small (...depending on what you're doing), When dealing with object/mixed dtypes When using the str/regex accessor functions Let's examine these situations individually. Iteration v/s Vectorization on Small Data Pandas follows a "Convention Over Configuration" approach in its API design. This means that the same API has been fitted to cater to a broad range of data and use cases. When a pandas function is called, the following things (among others) must internally be handled by the function, to ensure working Index/axis alignment Handling mixed datatypes Handling missing data Almost every function will have to deal with these to varying extents, and this presents an overhead. The overhead is less for numeric functions (for example, Series.add), while it is more pronounced for string functions (for example, Series.str.replace). for loops, on the other hand, are faster then you think. What's even better is list comprehensions (which create lists through for loops) are even faster as they are optimized iterative mechanisms for list creation. List comprehensions follow the pattern [f(x) for x in seq] Where seq is a pandas series or DataFrame column. Or, when operating over multiple columns, [f(x, y) for x, y in zip(seq1, seq2)] Where seq1 and seq2 are columns. Numeric Comparison Consider a simple boolean indexing operation. The list comprehension method has been timed against Series.ne (!=) and query. Here are the functions: # Boolean indexing with Numeric value comparison. df[df.A != df.B] # vectorized != df.query('A != B') # query (numexpr) df[[x != y for x, y in zip(df.A, df.B)]] # list comp For simplicity, I have used the perfplot package to run all the timeit tests in this post. The timings for the operations above are below: The list comprehension outperforms query for moderately sized N, and even outperforms the vectorized not equals comparison for tiny N. Unfortunately, the list comprehension scales linearly, so it does not offer much performance gain for larger N. Note It is worth mentioning that much of the benefit of list comprehension come from not having to worry about the index alignment, but this means that if your code is dependent on indexing alignment, this will break. In some cases, vectorised operations over the underlying NumPy arrays can be considered as bringing in the "best of both worlds", allowing for vectorisation without all the unneeded overhead of the pandas functions. This means that you can rewrite the operation above as df[df.A.values != df.B.values] Which outperforms both the pandas and list comprehension equivalents: NumPy vectorization is out of the scope of this post, but it is definitely worth considering, if performance matters. Value Counts Taking another example - this time, with another vanilla python construct that is faster than a for loop - collections.Counter. A common requirement is to compute the value counts and return the result as a dictionary. This is done with value_counts, np.unique, and Counter: # Value Counts comparison. ser.value_counts(sort=False).to_dict() # value_counts dict(zip(*np.unique(ser, return_counts=True))) # np.unique Counter(ser) # Counter The results are more pronounced, Counter wins out over both vectorized methods for a larger range of small N (~3500). Note More trivia (courtesy @user2357112). The Counter is implemented with a C accelerator, so while it still has to work with python objects instead of the underlying C datatypes, it is still faster than a for loop. Python power! Of course, the take away from here is that the performance depends on your data and use case. The point of these examples is to convince you not to rule out these solutions as legitimate options. If these still don't give you the performance you need, there is always cython and numba. Let's add this test into the mix. from numba import njit, prange @njit(parallel=True) def get_mask(x, y): result = [False] * len(x) for i in prange(len(x)): result[i] = x[i] != y[i] return np.array(result) df[get_mask(df.A.values, df.B.values)] # numba Numba offers JIT compilation of loopy python code to very powerful vectorized code. Understanding how to make numba work involves a learning curve. Operations with Mixed/object dtypes String-based Comparison Revisiting the filtering example from the first section, what if the columns being compared are strings? Consider the same 3 functions above, but with the input DataFrame cast to string. # Boolean indexing with string value comparison. df[df.A != df.B] # vectorized != df.query('A != B') # query (numexpr) df[[x != y for x, y in zip(df.A, df.B)]] # list comp So, what changed? The thing to note here is that string operations are inherently difficult to vectorize. Pandas treats strings as objects, and all operations on objects fall back to a slow, loopy implementation. Now, because this loopy implementation is surrounded by all the overhead mentioned above, there is a constant magnitude difference between these solutions, even though they scale the same. When it comes to operations on mutable/complex objects, there is no comparison. List comprehension outperforms all operations involving dicts and lists. Accessing Dictionary Value(s) by Key Here are timings for two operations that extract a value from a column of dictionaries: map and the list comprehension. The setup is in the Appendix, under the heading "Code Snippets". # Dictionary value extraction. ser.map(operator.itemgetter('value')) # map pd.Series([x.get('value') for x in ser]) # list comprehension Positional List Indexing Timings for 3 operations that extract the 0th element from a list of columns (handling exceptions), map, str.get accessor method, and the list comprehension: # List positional indexing. def get_0th(lst): try: return lst[0] # Handle empty lists and NaNs gracefully. except (IndexError, TypeError): return np.nan ser.map(get_0th) # map ser.str[0] # str accessor pd.Series([x[0] if len(x) > 0 else np.nan for x in ser]) # list comp pd.Series([get_0th(x) for x in ser]) # list comp safe Note If the index matters, you would want to do: pd.Series([...], index=ser.index) When reconstructing the series. List Flattening A final example is flattening lists. This is another common problem, and demonstrates just how powerful pure python is here. # Nested list flattening. pd.DataFrame(ser.tolist()).stack().reset_index(drop=True) # stack pd.Series(list(chain.from_iterable(ser.tolist()))) # itertools.chain pd.Series([y for x in ser for y in x]) # nested list comp Both itertools.chain.from_iterable and the nested list comprehension are pure python constructs, and scale much better than the stack solution. These timings are a strong indication of the fact that pandas is not equipped to work with mixed dtypes, and that you should probably refrain from using it to do so. Wherever possible, data should be present as scalar values (ints/floats/strings) in separate columns. Lastly, the applicability of these solutions depend widely on your data. So, the best thing to do would be to test these operations on your data before deciding what to go with. Notice how I have not timed apply on these solutions, because it would skew the graph (yes, it's that slow). Regex Operations, and .str Accessor Methods Pandas can apply regex operations such as str.contains, str.extract, and str.extractall, as well as other "vectorized" string operations (such as str.split, str.find, str.translate, and so on) on string columns. These functions are slower than list comprehensions, and are meant to be more convenience functions than anything else. It is usually much faster to pre-compile a regex pattern and iterate over your data with re.compile (also see Is it worth using Python's re.compile?). The list comp equivalent to str.contains looks something like this: p = re.compile(...) ser2 = pd.Series([x for x in ser if p.search(x)]) Or, ser2 = ser[[bool(p.search(x)) for x in ser]] If you need to handle NaNs, you can do something like ser[[bool(p.search(x)) if pd.notnull(x) else False for x in ser]] The list comp equivalent to str.extract (without groups) will look something like: df['col2'] = [p.search(x).group(0) for x in df['col']] If you need to handle no-matches and NaNs, you can use a custom function (still faster!): def matcher(x): m = p.search(str(x)) if m: return m.group(0) return np.nan df['col2'] = [matcher(x) for x in df['col']] The matcher function is very extensible. It can be fitted to return a list for each capture group, as needed. Just extract query the group or groups attribute of the matcher object. For str.extractall, change p.search to p.findall. String Extraction Consider a simple filtering operation. The idea is to extract 4 digits if it is preceded by an upper case letter. # Extracting strings. p = re.compile(r'(?<=[A-Z])(\d{4})') def matcher(x): m = p.search(x) if m: return m.group(0) return np.nan ser.str.extract(r'(?<=[A-Z])(\d{4})', expand=False) # str.extract pd.Series([matcher(x) for x in ser]) # list comprehension More Examples Full disclosure - I am the author (in part or whole) of these posts listed below. Fast punctuation removal with pandas String concatenation of two pandas columns Remove unwanted parts from strings in a column Replace all but the last occurrence of a character in a dataframe Conclusion As shown from the examples above, iteration shines when working with small rows of DataFrames, mixed datatypes, and regular expressions. The speedup you get depends on your data and your problem, so your mileage may vary. The best thing to do is to carefully run tests and see if the payout is worth the effort. The "vectorized" functions shine in their simplicity and readability, so if performance is not critical, you should definitely prefer those. Another side note, certain string operations deal with constraints that favour the use of NumPy. Here are two examples where careful NumPy vectorization outperforms python: Create new column with incremental values in a faster and efficient way - Answer by Divakar Fast punctuation removal with pandas - Answer by Paul Panzer Additionally, sometimes just operating on the underlying arrays via .values as opposed to on the Series or DataFrames can offer a healthy enough speedup for most usual scenarios (see the Note in the Numeric Comparison section above). So, for example df[df.A.values != df.B.values] would show instant performance boosts over df[df.A != df.B]. Using .values may not be appropriate in every situation, but it is a useful hack to know. As mentioned above, it's up to you to decide whether these solutions are worth the trouble of implementing. Appendix: Code Snippets import perfplot import operator import pandas as pd import numpy as np import re from collections import Counter from itertools import chain # Boolean indexing with Numeric value comparison. perfplot.show( setup=lambda n: pd.DataFrame(np.random.choice(1000, (n, 2)), columns=['A','B']), kernels=[ lambda df: df[df.A != df.B], lambda df: df.query('A != B'), lambda df: df[[x != y for x, y in zip(df.A, df.B)]], lambda df: df[get_mask(df.A.values, df.B.values)] ], labels=['vectorized !=', 'query (numexpr)', 'list comp', 'numba'], n_range=[2**k for k in range(0, 15)], xlabel='N' ) # Value Counts comparison. perfplot.show( setup=lambda n: pd.Series(np.random.choice(1000, n)), kernels=[ lambda ser: ser.value_counts(sort=False).to_dict(), lambda ser: dict(zip(*np.unique(ser, return_counts=True))), lambda ser: Counter(ser), ], labels=['value_counts', 'np.unique', 'Counter'], n_range=[2**k for k in range(0, 15)], xlabel='N', equality_check=lambda x, y: dict(x) == dict(y) ) # Boolean indexing with string value comparison. perfplot.show( setup=lambda n: pd.DataFrame(np.random.choice(1000, (n, 2)), columns=['A','B'], dtype=str), kernels=[ lambda df: df[df.A != df.B], lambda df: df.query('A != B'), lambda df: df[[x != y for x, y in zip(df.A, df.B)]], ], labels=['vectorized !=', 'query (numexpr)', 'list comp'], n_range=[2**k for k in range(0, 15)], xlabel='N', equality_check=None ) # Dictionary value extraction. ser1 = pd.Series([{'key': 'abc', 'value': 123}, {'key': 'xyz', 'value': 456}]) perfplot.show( setup=lambda n: pd.concat([ser1] * n, ignore_index=True), kernels=[ lambda ser: ser.map(operator.itemgetter('value')), lambda ser: pd.Series([x.get('value') for x in ser]), ], labels=['map', 'list comprehension'], n_range=[2**k for k in range(0, 15)], xlabel='N', equality_check=None ) # List positional indexing. ser2 = pd.Series([['a', 'b', 'c'], [1, 2], []]) perfplot.show( setup=lambda n: pd.concat([ser2] * n, ignore_index=True), kernels=[ lambda ser: ser.map(get_0th), lambda ser: ser.str[0], lambda ser: pd.Series([x[0] if len(x) > 0 else np.nan for x in ser]), lambda ser: pd.Series([get_0th(x) for x in ser]), ], labels=['map', 'str accessor', 'list comprehension', 'list comp safe'], n_range=[2**k for k in range(0, 15)], xlabel='N', equality_check=None ) # Nested list flattening. ser3 = pd.Series([['a', 'b', 'c'], ['d', 'e'], ['f', 'g']]) perfplot.show( setup=lambda n: pd.concat([ser2] * n, ignore_index=True), kernels=[ lambda ser: pd.DataFrame(ser.tolist()).stack().reset_index(drop=True), lambda ser: pd.Series(list(chain.from_iterable(ser.tolist()))), lambda ser: pd.Series([y for x in ser for y in x]), ], labels=['stack', 'itertools.chain', 'nested list comp'], n_range=[2**k for k in range(0, 15)], xlabel='N', equality_check=None ) # Extracting strings. ser4 = pd.Series(['foo xyz', 'test A1234', 'D3345 xtz']) perfplot.show( setup=lambda n: pd.concat([ser4] * n, ignore_index=True), kernels=[ lambda ser: ser.str.extract(r'(?<=[A-Z])(\d{4})', expand=False), lambda ser: pd.Series([matcher(x) for x in ser]) ], labels=['str.extract', 'list comprehension'], n_range=[2**k for k in range(0, 15)], xlabel='N', equality_check=None )


Instruction:
Attempted relative import with no known parent package

from ..box_utils import decode, nms This line is giving ImportError: attempted relative import with no known parent package What is this error and how to resolve it?

Output: Apparently, box_utils.py isn't part of a package. You still can import functions defined in this file, but only if the python script that tries to import these functions lives in the same directory as box_utils.py, see this answer. Nota bene: In my case, I stumbled upon this error with an import statement with one period, like this: from .foo import foo. This syntax, however, tells Python that foo.py is part of a package, which wasn't the case. The error disappeared when I removed the period.


Instruction:
how to convert an RGB image to numpy array?

I have an RGB image. I want to convert it to numpy array. I did the following im = cv.LoadImage("abc.tiff") a = numpy.asarray(im) It creates an array with no shape. I assume it is a iplimage object.

Output: You can use newer OpenCV python interface (if I'm not mistaken it is available since OpenCV 2.2). It natively uses numpy arrays: import cv2 im = cv2.imread("abc.tiff",mode='RGB') print(type(im)) result: <type 'numpy.ndarray'>


Instruction:
ImportError: cannot import name '...' from partially initialized module '...' (most likely due to a circular import)

I'm upgrading an application from Django 1.11.25 (Python 2.6) to Django 3.1.3 (Python 3.8.5) and, when I run manage.py makemigrations, I receive this message: File "/home/eduardo/projdevs/upgrade-intra/corporate/models/section.py", line 9, in <module> from authentication.models import get_sentinel** ImportError: cannot import name 'get_sentinel' from partially initialized module 'authentication.models' (most likely due to a circular import) (/home/eduardo/projdevs/upgrade-intra/authentication/models.py)** My models are: authentication / models.py from django.conf import settings from django.contrib.auth.models import AbstractUser, UserManager from django.db import models from django.db.models.signals import post_save from django.utils import timezone from corporate.constants import GROUP_SUPPORT from corporate.models import Phone, Room, Section from library.exceptions import ErrorMessage from library.model import update_through_dict from .constants import INTERNAL_USER, EXTERNAL_USER, SENTINEL_USERNAME, SPECIAL_USER, USER_TYPES_DICT class UserProfile(models.Model): user = models.OneToOneField( 'User', on_delete=models.CASCADE, unique=True, db_index=True ) ... phone = models.ForeignKey('corporate.Phone', on_delete=models.SET_NULL, ...) room = models.ForeignKey('corporate.Room', on_delete=models.SET_NULL, ...) section = models.ForeignKey('corporate.Section', on_delete=models.SET_NULL, ...) objects = models.Manager() ... class CustomUserManager(UserManager): def __init__(self, type=None): super(CustomUserManager, self).__init__() self.type = type def get_queryset(self): qs = super(CustomUserManager, self).get_queryset() if self.type: qs = qs.filter(type=self.type).order_by('first_name', 'last_name') return qs def get_this_types(self, types): qs = super(CustomUserManager, self).get_queryset() qs = qs.filter(type__in=types).order_by('first_name', 'last_name') return qs def get_all_excluding(self, types): qs = super(CustomUserManager, self).get_queryset() qs = qs.filter(~models.Q(type__in=types)).order_by('first_name', 'last_name') return qs class User(AbstractUser): type = models.PositiveIntegerField('...', default=SPECIAL_USER) username = models.CharField('...', max_length=256, unique=True) first_name = models.CharField('...', max_length=40, blank=True) last_name = models.CharField('...', max_length=80, blank=True) date_joined = models.DateTimeField('...', default=timezone.now) previous_login = models.DateTimeField('...', default=timezone.now) objects = CustomUserManager() ... def get_profile(self): if self.type == INTERNAL_USER: ... return None def get_or_create_profile(self): profile = self.get_profile() if not profile and self.type == INTERNAL_USER: ... return profile def update(self, changes): ... class ExternalUserProxy(User): objects = CustomUserManager(type=EXTERNAL_USER) class Meta: proxy = True verbose_name = '...' verbose_name_plural = '...' class InternalUserProxy(User): objects = CustomUserManager(type=INTERNAL_USER) class Meta: proxy = True verbose_name = '...' verbose_name_plural = '...' def create_profile(sender, instance, created, **kwargs): if created and instance.type == INTERNAL_USER: try: profile = UserProfile() profile.user = instance profile.save() except: pass post_save.connect(create_profile, sender=User) def get_sentinel(): try: sentinel = User.objects.get(username__exact=SENTINEL_USERNAME) except User.DoesNotExist: settings.LOGGER.error("...") from django.contrib.auth.models import Group sentinel = User() sentinel.username = SENTINEL_USERNAME sentinel.first_name = "..." sentinel.last_name = "..." sentinel.set_unusable_password() sentinel.save() technical = Group.objects.get(name=GROUP_SUPPORT) sentinel = User.objects.get(username__exact=SENTINEL_USERNAME) sentinel.groups.add(technical) sentinel.save() return sentinel corporate / models / __init__.py ... from .section import Section ... corporate / models / section.py from django.conf import settings from authentication.models import get_sentinel from .room import Room class Section(models.Model): ... boss = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.SET(get_sentinel), ...) surrogate = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.SET(get_sentinel), ...) room = models.ForeignKey(Room, on_delete=models.SET_NULL, ...) is_subordinate_to = models.ForeignKey('self', on_delete=models.SET_NULL, ...) ... What am I doing wrong?

Output: You have a circular import. authentication/models imports corporate/models, which imports corporate/models/section, which imports authentication/models. You can't do that. Rewrite and/or rearrange your modules so that circular imports aren't needed. One strategy to do this is to organize your modules into a hierarchy, and make sure that a module only imports other modules that are lower in the hierarchy. (This hierarchy can be an actual directory structure, but it doesn't have to be; it can just be a mental note in the programmer's mind.)


Instruction:
How to move a model between two Django apps (Django 1.7)

So about a year ago I started a project and like all new developers I didn't really focus too much on the structure, however now I am further along with Django it has started to appear that my project layout mainly my models are horrible in structure. I have models mainly held in a single app and really most of these models should be in their own individual apps, I did try and resolve this and move them with south however I found it tricky and really difficult due to foreign keys ect. However due to Django 1.7 and built in support for migrations is there a better way to do this now?

Output: I am removing the old answer as may result in data loss. As ozan mentioned, we can create 2 migrations one in each app. The comments below this post refer to my old answer. First migration to remove model from 1st app. $ python manage.py makemigrations old_app --empty Edit migration file to include these operations. class Migration(migrations.Migration): database_operations = [migrations.AlterModelTable('TheModel', 'newapp_themodel')] state_operations = [migrations.DeleteModel('TheModel')] operations = [ migrations.SeparateDatabaseAndState( database_operations=database_operations, state_operations=state_operations) ] Second migration which depends on first migration and create the new table in 2nd app. After moving model code to 2nd app $ python manage.py makemigrations new_app and edit migration file to something like this. class Migration(migrations.Migration): dependencies = [ ('old_app', 'above_migration') ] state_operations = [ migrations.CreateModel( name='TheModel', fields=[ ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)), ], options={ 'db_table': 'newapp_themodel', }, bases=(models.Model,), ) ] operations = [ migrations.SeparateDatabaseAndState(state_operations=state_operations) ]


Instruction:
How to extract a floating number from a string

I have a number of strings similar to Current Level: 13.4 db. and I would like to extract just the floating point number. I say floating and not decimal as it's sometimes whole. Can RegEx do this or is there a better way?

Output: If your float is always expressed in decimal notation something like >>> import re >>> re.findall("\d+\.\d+", "Current Level: 13.4db.") ['13.4'] may suffice. A more robust version would be: >>> re.findall(r"[-+]?(?:\d*\.*\d+)", "Current Level: -13.2db or 14.2 or 3") ['-13.2', '14.2', '3'] If you want to validate user input, you could alternatively also check for a float by stepping to it directly: user_input = "Current Level: 1e100 db" for token in user_input.split(): try: # if this succeeds, you have your (first) float print(float(token), "is a float") except ValueError: print(token, "is something else") # => Would print ... # # Current is something else # Level: is something else # 1e+100 is a float # db is something else


Instruction:
Local (?) variable referenced before assignment

test1 = 0 def test_func(): test1 += 1 test_func() I am receiving the following error: UnboundLocalError: local variable 'test1' referenced before assignment. Error says that 'test1' is local variable but i thought that this variable is global So is it global or local and how to solve this error without passing global test1 as argument to test_func?

Output: In order for you to modify test1 while inside a function you will need to do define test1 as a global variable, for example: test1 = 0 def test_func(): global test1 test1 += 1 test_func() However, if you only need to read the global variable you can print it without using the keyword global, like so: test1 = 0 def test_func(): print(test1) test_func() But whenever you need to modify a global variable you must use the keyword global.


Instruction:
Should I use name mangling in Python?

In other languages, a general guideline that helps produce better code is always make everything as hidden as possible. If in doubt about whether a variable should be private or protected, it's better to go with private. Does the same hold true for Python? Should I use two leading underscores on everything at first, and only make them less hidden (only one underscore) as I need them? If the convention is to use only one underscore, I'd also like to know the rationale. Here's a comment I left on JBernardo's answer. It explains why I asked this question and also why I'd like to know why Python is different from the other languages: I come from languages that train you to think everything should be only as public as needed and no more. The reasoning is that this will reduce dependencies and make the code safer to alter. The Python way of doing things in reverse -- starting from public and going towards hidden -- is odd to me.

Output: When in doubt, leave it "public" - I mean, do not add anything to obscure the name of your attribute. If you have a class with some internal value, do not bother about it. Instead of writing: class Stack(object): def __init__(self): self.__storage = [] # Too uptight def push(self, value): self.__storage.append(value) write this by default: class Stack(object): def __init__(self): self.storage = [] # No mangling def push(self, value): self.storage.append(value) This is for sure a controversial way of doing things. Python newbies hate it, and even some old Python guys despise this default - but it is the default anyway, so I recommend you to follow it, even if you feel uncomfortable. If you really want to send the message "Can't touch this!" to your users, the usual way is to precede the variable with one underscore. This is just a convention, but people understand it and take double care when dealing with such stuff: class Stack(object): def __init__(self): self._storage = [] # This is ok, but Pythonistas use it to be relaxed about it def push(self, value): self._storage.append(value) This can be useful, too, for avoiding conflict between property names and attribute names: class Person(object): def __init__(self, name, age): self.name = name self._age = age if age >= 0 else 0 @property def age(self): return self._age @age.setter def age(self, age): if age >= 0: self._age = age else: self._age = 0 What about the double underscore? Well, we use the double underscore magic mainly to avoid accidental overloading of methods and name conflicts with superclasses' attributes. It can be pretty valuable if you write a class to be extended many times. If you want to use it for other purposes, you can, but it is neither usual nor recommended. EDIT: Why is this so? Well, the usual Python style does not emphasize making things private - on the contrary! There are many reasons for that - most of them controversial... Let us see some of them. Python has properties Today, most OO languages use the opposite approach: what should not be used should not be visible, so attributes should be private. Theoretically, this would yield more manageable, less coupled classes because no one would change the objects' values recklessly. However, it is not so simple. For example, Java classes have many getters that only get the values and setters that only set the values. You need, let us say, seven lines of code to declare a single attribute - which a Python programmer would say is needlessly complex. Also, you write a lot of code to get one public field since you can change its value using the getters and setters in practice. So why follow this private-by-default policy? Just make your attributes public by default. Of course, this is problematic in Java because if you decide to add some validation to your attribute, it would require you to change all: person.age = age; in your code to, let us say, person.setAge(age); setAge() being: public void setAge(int age) { if (age >= 0) { this.age = age; } else { this.age = 0; } } So in Java (and other languages), the default is to use getters and setters anyway because they can be annoying to write but can spare you much time if you find yourself in the situation I've described. However, you do not need to do it in Python since Python has properties. If you have this class: class Person(object): def __init__(self, name, age): self.name = name self.age = age ...and then you decide to validate ages, you do not need to change the person.age = age pieces of your code. Just add a property (as shown below) class Person(object): def __init__(self, name, age): self.name = name self._age = age if age >= 0 else 0 @property def age(self): return self._age @age.setter def age(self, age): if age >= 0: self._age = age else: self._age = 0 Suppose you can do it and still use person.age = age, why would you add private fields and getters and setters? (Also, see Python is not Java and this article about the harms of using getters and setters.). Everything is visible anyway - and trying to hide complicates your work Even in languages with private attributes, you can access them through some reflection/introspection library. And people do it a lot, in frameworks and for solving urgent needs. The problem is that introspection libraries are just a complicated way of doing what you could do with public attributes. Since Python is a very dynamic language, adding this burden to your classes is counterproductive. The problem is not being possible to see - it is being required to see For a Pythonista, encapsulation is not the inability to see the internals of classes but the possibility of avoiding looking at it. Encapsulation is the property of a component that the user can use without concerning about the internal details. If you can use a component without bothering yourself about its implementation, then it is encapsulated (in the opinion of a Python programmer). Now, if you wrote a class you can use it without thinking about implementation details, there is no problem if you want to look inside the class for some reason. The point is: your API should be good, and the rest is details. Guido said so Well, this is not controversial: he said so, actually. (Look for "open kimono.") This is culture Yes, there are some reasons, but no critical reason. This is primarily a cultural aspect of programming in Python. Frankly, it could be the other way, too - but it is not. Also, you could just as easily ask the other way around: why do some languages use private attributes by default? For the same main reason as for the Python practice: because it is the culture of these languages, and each choice has advantages and disadvantages. Since there already is this culture, you are well-advised to follow it. Otherwise, you will get annoyed by Python programmers telling you to remove the __ from your code when you ask a question in Stack Overflow :)


Instruction:
How can I check if a module has been imported?

How do I check if I imported a module somewhere in the code? if not has_imported("somemodule"): print('you have not imported somemodule') The reason that I would like to check if I already imported a module is because I have a module that I don't want to import, because sometimes it messes up my program.

Output: Test for the module name in the sys.modules dictionary: import sys modulename = 'datetime' if modulename not in sys.modules: print 'You have not imported the {} module'.format(modulename) From the documenation: This is a dictionary that maps module names to modules which have already been loaded. Note that an import statement does two things: if the module has never been imported before (== not present in sys.modules), then it is loaded and added to sys.modules. Bind 1 or more names in the current namespace that reference the module object or to objects that are members of the module namespace. The expression modulename not in sys.modules tests if step 1 has taken place. Testing for the result of step 2 requires knowing what exact import statement was used as they set different names to reference different objects: import modulename sets modulename = sys.modules['modulename'] import packagename.nestedmodule sets packagename = sys.modules['packagename'] (no matter how many addional levels you add) import modulename as altname sets altname = sys.module['modulename'] import packagename.nestedmodule as altname sets altname = sys.modules['packagename.nestedmodule'] from somemodule import objectname sets objectname = sys.modules['somemodule'].objectname from packagename import nestedmodulename sets nestedmodulename = sys.modules['packagename.nestedmodulename'] (only when there was no object named nestedmodulename in the packagename namespace before this import, an additional name for the nested module is added to the parent package namespace at this point) from somemodule import objectname as altname sets altname = sys.modules['somemodule'].objectname from packagename import nestedmodulename as altname sets altname = sys.modules['packagename.nestedmodulename'] (only when there was no object named nestedmodulename in the packagename namespace before this import, an additional name for the nested module is added to the parent package namespace at this point) You can test if the name to which the imported object was bound exists in a given namespace: # is this name visible in the current scope: 'importedname' in dir() # or, is this a name in the globals of the current module: 'importedname' in globals() # or, does the name exist in the namespace of another module: 'importedname' in globals(sys.modules['somemodule']) This only tells you of the name exists (has been bound), not if it refers to a specific module or object from that module. You could further introspect that object or test if it’s the same object as what’s available in sys.modules, if you need to rule out that the name has been set to something else entirely since then.


Instruction:
How to avoid explicit 'self' in Python?

I have been learning Python by following some pygame tutorials. Therein I found extensive use of the keyword self, and coming from a primarily Java background, I find that I keep forgetting to type self. For example, instead of self.rect.centerx I would type rect.centerx, because, to me, rect is already a member variable of the class. The Java parallel I can think of for this situation is having to prefix all references to member variables with this. Am I stuck prefixing all member variables with self, or is there a way to declare them that would allow me to avoid having to do so? Even if what I am suggesting isn't pythonic, I'd still like to know if it is possible. I have taken a look at these related SO questions, but they don't quite answer what I am after: Python - why use “self” in a class? Why do you need explicitly have the “self” argument into a Python method?

Output: In Java terms: Python doesn't have member functions, all class functions are static, and are called with a reference to the actual class instance as first argument when invoked as member function. This means that when your code has a class MyClass and you build an instance m = MyClass(), calling m.do_something() will be executed as MyClass.do_something(m). Also note that this first argument can technically be called anything you want, but the convention is to use self, and you should stick to that convention if you want others (including your future self) to be able to easily read your code. The result is there's never any confusion over what's a member and what's not, even without the full class definition visible. This leads to useful properties, such as: you can't add members which accidentally shadow non-members and thereby break code. One extreme example: you can write a class without any knowledge of what base classes it might have, and always know whether you are accessing a member or not: class A(some_function()): def f(self): self.member = 42 self.method() That's the complete code! (some_function returns the type used as a base.) Another, where the methods of a class are dynamically composed: class B(object): pass print B() # <__main__.B object at 0xb7e4082c> def B_init(self): self.answer = 42 def B_str(self): return "<The answer is %s.>" % self.answer # notice these functions require no knowledge of the actual class # how hard are they to read and realize that "members" are used? B.__init__ = B_init B.__str__ = B_str print B() # <The answer is 42.> Remember, both of these examples are extreme and you won't see them every day, nor am I suggesting you should often write code like this, but they do clearly show aspects of self being explicitly required.


Instruction:
How can I save an image with PIL?

I have just done some image processing using the Python image library (PIL) using a post I found earlier to perform fourier transforms of images and I can't get the save function to work. The whole code works fine but it just wont save the resulting image: from PIL import Image import numpy as np i = Image.open("C:/Users/User/Desktop/mesh.bmp") i = i.convert("L") a = np.asarray(i) b = np.abs(np.fft.rfft2(a)) j = Image.fromarray(b) j.save("C:/Users/User/Desktop/mesh_trans",".bmp") The error I get is the following: save_handler = SAVE[string.upper(format)] # unknown format KeyError: '.BMP' How can I save an image with Pythons PIL?

Output: The error regarding the file extension has been handled, you either use BMP (without the dot) or pass the output name with the extension already. Now to handle the error you need to properly modify your data in the frequency domain to be saved as an integer image, PIL is telling you that it doesn't accept float data to save as BMP. Here is a suggestion (with other minor modifications, like using fftshift and numpy.array instead of numpy.asarray) for doing the conversion for proper visualization: import sys import numpy from PIL import Image img = Image.open(sys.argv[1]).convert('L') im = numpy.array(img) fft_mag = numpy.abs(numpy.fft.fftshift(numpy.fft.fft2(im))) visual = numpy.log(fft_mag) visual = (visual - visual.min()) / (visual.max() - visual.min()) result = Image.fromarray((visual * 255).astype(numpy.uint8)) result.save('out.bmp')


Instruction:
How do I read text from the Windows clipboard in Python?

How do I read text from the (windows) clipboard with python?

Output: You can use the module called win32clipboard, which is part of pywin32. Here is an example that first sets the clipboard data then gets it: import win32clipboard # set clipboard data win32clipboard.OpenClipboard() win32clipboard.EmptyClipboard() win32clipboard.SetClipboardText('testing 123') win32clipboard.CloseClipboard() # get clipboard data win32clipboard.OpenClipboard() data = win32clipboard.GetClipboardData() win32clipboard.CloseClipboard() print data An important reminder from the documentation: When the window has finished examining or changing the clipboard, close the clipboard by calling CloseClipboard. This enables other windows to access the clipboard. Do not place an object on the clipboard after calling CloseClipboard.


Instruction:
Running Python code in Vim

I am writing Python code using Vim, and every time I want to run my code, I type this inside Vim: :w !python This gets frustrating, so I was looking for a quicker method to run Python code inside Vim. Executing Python scripts from a terminal maybe? I am using Linux.

Output: How about adding an autocmd to your ~/.vimrc-file, creating a mapping: autocmd FileType python map <buffer> <F9> :w<CR>:exec '!python3' shellescape(@%, 1)<CR> autocmd FileType python imap <buffer> <F9> <esc>:w<CR>:exec '!python3' shellescape(@%, 1)<CR> then you could press <F9> to execute the current buffer with python Explanation: autocmd: command that Vim will execute automatically on {event} (here: if you open a python file) [i]map: creates a keyboard shortcut to <F9> in insert/normal mode <buffer>: If multiple buffers/files are open: just use the active one <esc>: leaving insert mode :w<CR>: saves your file !: runs the following command in your shell (try :!ls) %: is replaced by the filename of your active buffer. But since it can contain things like whitespace and other "bad" stuff it is better practise not to write :python %, but use: shellescape: escape the special characters. The 1 means with a backslash TL;DR: The first line will work in normal mode and once you press <F9> it first saves your file and then run the file with python. The second does the same thing, but leaves insert mode first


Instruction:
How do I add custom field to Python log format string?

My current format string is: formatter = logging.Formatter('%(asctime)s : %(message)s') and I want to add a new field called app_name which will have a different value in each script that contains this formatter. import logging formatter = logging.Formatter('%(asctime)s %(app_name)s : %(message)s') syslog.setFormatter(formatter) logger.addHandler(syslog) But I'm not sure how to pass that app_name value to the logger to interpolate into the format string. I can obviously get it to appear in the log message by passing it each time but this is messy. I've tried: logging.info('Log message', app_name='myapp') logging.info('Log message', {'app_name', 'myapp'}) logging.info('Log message', 'myapp') but none work.

Output: Python3 As of Python3.2 you can now use LogRecordFactory import logging logging.basicConfig(format="%(custom_attribute)s - %(message)s") old_factory = logging.getLogRecordFactory() def record_factory(*args, **kwargs): record = old_factory(*args, **kwargs) record.custom_attribute = "my-attr" return record logging.setLogRecordFactory(record_factory) >>> logging.info("hello") my-attr - hello Of course, record_factory can be customized to be any callable and the value of custom_attribute could be updated if you keep a reference to the factory callable. Why is that better than using Adapters / Filters? You do not need to pass your logger around the application It actually works with 3rd party libraries that use their own logger (by just calling logger = logging.getLogger(..)) would now have the same log format. (this is not the case with Filters / Adapters where you need to be using the same logger object) You can stack/chain multiple factories


Instruction:
Haversine formula in Python (bearing and distance between two GPS points)

Problem I would like to know how to get the distance and bearing between two GPS points. I have researched on the haversine distance. Someone told me that I could also find the bearing using the same data. Everything is working fine, but the bearing doesn't quite work right yet. The bearing outputs negative, but it should be between 0 - 360 degrees. The set data should make the horizontal bearing 96.02166666666666 and is: Start point: 53.32055555555556, -1.7297222222222221 Bearing: 96.02166666666666 Distance: 2 km Destination point: 53.31861111111111, -1.6997222222222223 Final bearing: 96.04555555555555 Here is my new code: from math import * Aaltitude = 2000 Oppsite = 20000 lat1 = 53.32055555555556 lat2 = 53.31861111111111 lon1 = -1.7297222222222221 lon2 = -1.6997222222222223 lon1, lat1, lon2, lat2 = map(radians, [lon1, lat1, lon2, lat2]) dlon = lon2 - lon1 dlat = lat2 - lat1 a = sin(dlat/2)**2 + cos(lat1) * cos(lat2) * sin(dlon/2)**2 c = 2 * atan2(sqrt(a), sqrt(1-a)) Base = 6371 * c Bearing = atan2(cos(lat1)*sin(lat2)-sin(lat1)*cos(lat2)*cos(lon2-lon1), sin(lon2-lon1)*cos(lat2)) Bearing = degrees(Bearing) print "" print "" print "--------------------" print "Horizontal Distance: " print Base print "--------------------" print "Bearing: " print Bearing print "--------------------" Base2 = Base * 1000 distance = Base * 2 + Oppsite * 2 / 2 Caltitude = Oppsite - Aaltitude a = Oppsite/Base b = atan(a) c = degrees(b) distance = distance / 1000 print "The degree of vertical angle is: " print c print "--------------------" print "The distance between the Balloon GPS and the Antenna GPS is: " print distance print "--------------------"

Output: Here's a Python version: from math import radians, cos, sin, asin, sqrt def haversine(lon1, lat1, lon2, lat2): """ Calculate the great circle distance in kilometers between two points on the earth (specified in decimal degrees) """ # convert decimal degrees to radians lon1, lat1, lon2, lat2 = map(radians, [lon1, lat1, lon2, lat2]) # haversine formula dlon = lon2 - lon1 dlat = lat2 - lat1 a = sin(dlat/2)**2 + cos(lat1) * cos(lat2) * sin(dlon/2)**2 c = 2 * asin(sqrt(a)) r = 6371 # Radius of earth in kilometers. Use 3956 for miles. Determines return value units. return c * r


Instruction:
How to trace the path in a Breadth-First Search?

How do you trace the path of a Breadth-First Search, such that in the following example: If searching for key 11, return the shortest list connecting 1 to 11. [1, 4, 7, 11]

Output: You should have look at http://en.wikipedia.org/wiki/Breadth-first_search first. Below is a quick implementation, in which I used a list of list to represent the queue of paths. # graph is in adjacent list representation graph = { '1': ['2', '3', '4'], '2': ['5', '6'], '5': ['9', '10'], '4': ['7', '8'], '7': ['11', '12'] } def bfs(graph, start, end): # maintain a queue of paths queue = [] # push the first path into the queue queue.append([start]) while queue: # get the first path from the queue path = queue.pop(0) # get the last node from the path node = path[-1] # path found if node == end: return path # enumerate all adjacent nodes, construct a # new path and push it into the queue for adjacent in graph.get(node, []): new_path = list(path) new_path.append(adjacent) queue.append(new_path) print bfs(graph, '1', '11') This prints: ['1', '4', '7', '11'] Another approach would be maintaining a mapping from each node to its parent, and when inspecting the adjacent node, record its parent. When the search is done, simply backtrace according the parent mapping. graph = { '1': ['2', '3', '4'], '2': ['5', '6'], '5': ['9', '10'], '4': ['7', '8'], '7': ['11', '12'] } def backtrace(parent, start, end): path = [end] while path[-1] != start: path.append(parent[path[-1]]) path.reverse() return path def bfs(graph, start, end): parent = {} queue = [] queue.append(start) while queue: node = queue.pop(0) if node == end: return backtrace(parent, start, end) for adjacent in graph.get(node, []): if node not in queue : parent[adjacent] = node # <<<<< record its parent queue.append(adjacent) print bfs(graph, '1', '11') The above codes are based on the assumption that there's no cycles.


Instruction:
Get raw POST body in Python Flask regardless of Content-Type header

Previously, I asked How to get data received in Flask request because request.data was empty. The answer explained that request.data is the raw post body, but will be empty if form data is parsed. How can I get the raw post body unconditionally? @app.route('/', methods=['POST']) def parse_request(): data = request.data # empty in some cases # always need raw data here, not parsed form data

Output: Use request.get_data() to get the raw data, regardless of content type. The data is cached and you can subsequently access request.data, request.json, request.form at will. If you access request.data first, it will call get_data with an argument to parse form data first. If the request has a form content type (multipart/form-data, application/x-www-form-urlencoded, or application/x-url-encoded) then the raw data will be consumed. request.data and request.json will appear empty in this case.


Instruction:
NumPy selecting specific column index per row by using a list of indexes

I'm struggling to select the specific columns per row of a NumPy matrix. Suppose I have the following matrix which I would call X: [1, 2, 3] [4, 5, 6] [7, 8, 9] I also have a list of column indexes per every row which I would call Y: [1, 0, 2] I need to get the values: [2] [4] [9] Instead of a list with indexes Y, I can also produce a matrix with the same shape as X where every column is a bool / int in the range 0-1 value, indicating whether this is the required column. [0, 1, 0] [1, 0, 0] [0, 0, 1] I know this can be done with iterating over the array and selecting the column values I need. However, this will be executed frequently on big arrays of data and that's why it has to run as fast as it can. I was thus wondering if there is a better solution?

Output: If you've got a boolean array you can do direct selection based on that like so: >>> a = np.array([True, True, True, False, False]) >>> b = np.array([1,2,3,4,5]) >>> b[a] array([1, 2, 3]) To go along with your initial example you could do the following: >>> a = np.array([[1,2,3], [4,5,6], [7,8,9]]) >>> b = np.array([[False,True,False],[True,False,False],[False,False,True]]) >>> a[b] array([2, 4, 9]) You can also add in an arange and do direct selection on that, though depending on how you're generating your boolean array and what your code looks like YMMV. >>> a = np.array([[1,2,3], [4,5,6], [7,8,9]]) >>> a[np.arange(len(a)), [1,0,2]] array([2, 4, 9])


Instruction:
Copy file or directories recursively in Python

Python seems to have functions for copying files (e.g. shutil.copy) and functions for copying directories (e.g. shutil.copytree) but I haven't found any function that handles both. Sure, it's trivial to check whether you want to copy a file or a directory, but it seems like a strange omission. Is there really no standard function that works like the unix cp -r command, i.e. supports both directories and files and copies recursively? What would be the most elegant way to work around this problem in Python?

Output: I suggest you first call shutil.copytree, and if an exception is thrown, then retry with shutil.copy. import shutil, errno def copyanything(src, dst): try: shutil.copytree(src, dst) except OSError as exc: # python >2.5 if exc.errno in (errno.ENOTDIR, errno.EINVAL): shutil.copy(src, dst) else: raise


Instruction:
Remove trailing newline from the elements of a string list

I have to take a large list of words in the form: ['this\n', 'is\n', 'a\n', 'list\n', 'of\n', 'words\n'] and then using the strip function, turn it into: ['this', 'is', 'a', 'list', 'of', 'words'] I thought that what I had written would work, but I keep getting an error saying: "'list' object has no attribute 'strip'" Here is the code that I tried: strip_list = [] for lengths in range(1,20): strip_list.append(0) #longest word in the text file is 20 characters long for a in lines: strip_list.append(lines[a].strip())

Output: You can either use a list comprehension my_list = ['this\n', 'is\n', 'a\n', 'list\n', 'of\n', 'words\n'] stripped = [s.strip() for s in my_list] or alternatively use map(): stripped = list(map(str.strip, my_list)) In Python 2, map() directly returned a list, so you didn't need the call to list. In Python 3, the list comprehension is more concise and generally considered more idiomatic.


Instruction:
Hide tick label values but keep axis labels

I have this image: plt.plot(sim_1['t'],sim_1['V'],'k') plt.ylabel('V') plt.xlabel('t') plt.show() I want to hide the numbers; if I use: plt.axis('off') ...I get this image: It also hide the labels, V and t. How can I keep the labels while hiding the values?

Output: If you use the matplotlib object-oriented approach, this is a simple task using ax.set_xticklabels() and ax.set_yticklabels(). Here we can just set them to an empty list to remove any labels: import matplotlib.pyplot as plt # Create Figure and Axes instances fig,ax = plt.subplots(1) # Make your plot, set your axes labels ax.plot(sim_1['t'],sim_1['V'],'k') ax.set_ylabel('V') ax.set_xlabel('t') # Turn off tick labels ax.set_yticklabels([]) ax.set_xticklabels([]) plt.show() If you also want to remove the tick marks as well as the labels, you can use ax.set_xticks() and ax.set_yticks() and set those to an empty list as well: ax.set_xticks([]) ax.set_yticks([])