A Journey in Data & Music

Understanding Gain Staging & Gain Structure

Tue, 13 Apr 2021 17:20:00 +0000

Preface

Following post is adapted from my graduation thesis in SAE Audio Engineering programme.

I continually see questions and confusion regarding gain staging, and this document is still relevant even though I wrote it in 2012. While some parts of it regarding analog setups may not have the same level of appeal anymore, it’s still important to know where we are coming from even working fully inside the box.

Enjoy!

Introduction

Digital Summing, High Dynamic Range, In-the-Box Mixing, HD Audio, 128x Oversampling, 64 bit Internal Processing…

We hear all these words nowadays, and we usually take them for granted, as these concepts refer to things that are happening in the background while we’re recording or mixing our audio materials. We seem to be free from the meticulous fiddling, calibration and gain structuring of analog equipment, as our dynamic ranges gets higher and noise floors gets lower.

However, even the most “digital” equipment have an optimum working range and in order to fully utilize the wide dynamic range that modern equipments provide us, proper gain structuring remains a vital part of our workflow. This is also important as 99% of all music album sales are CDs or Digital Formats¹ which requires reducing the dynamic range, while still maintaining an acceptable level of noise floor.

In the light of these information, this research paper aims to determine the best possible practices regarding gain structure in various systems, by referring to fundamental concepts that affect gain structure and our perception of gain such as bit depth, dynamic range, headroom, metering, decibels, and calibration.

Fundamental Concepts

In order to understand gain structuring in all its depth, one should have the necessary knowledge of the elements that deals with gain, audio levels, and differences and similarities between digital & analog audio.

Also, benefits of attaining a proper gain structure or problems that can arise with improper level adjustments must be fully comprehended so that the important question of “why?” can be answered easily.

Components of Signal Flow

A typical Audio Signal Flow regarding gain structure consists of the following components:

Figure 1: Typical Audio Signal Flow

Signal Source: This is the original sound we want to process. This may be a guitar, a human voice, a digital signal generator; basically anything that produces a sound can be our signal source. In terms of gain structuring, our ability to modify the amplitude of a signal source is limited (unless the source is purely digital), as this also often changes the natural timbre and tone of the signal. Think of a vocalist keeping his/her volume down to control gain, it’s not natural.
Transducer: This is the device that converts the sound pressure waves (acoustical energy) of acoustical signal sources into electrical signals (or vice versa). Microphones, pick-ups, speakers and headphones are primary examples for this. Gain control in transducers are also limited as microphones or other transducers offer very few controls regarding gain structure.
Preamp: Preamps are devices that take low-level signals from transducers and converts them to line-level signals so the succeeding equipment can process the signal. Preamps are the primary and most important components in achieving proper gain structure. If the level of a signal coming from a transducer is not adjusted properly at the gain stage of the preamp, it may be too low or too high at its output, which affects the operation of all the succeeding components. The popular saying “Garbage in, garbage out” fits especially well here.
A/D & D/A Converters: Now, if at some point, the signal is wanted to be processed in the digital domain, we must talk about Analog-to-Digital conversion. To do this, the continuous analog signal must be sampled and approximated by frequency and amplitude, and converted to numbers that represent them. From gain structure perspective, the most important aspect of the converter is its usable dynamic range, or its SNR (Signal-to-Noise Ratio). To utilize the full dynamic range and lower the noise floor the converter has to offer, the level of the signal going into the converter, as well as coming out of the preceding equipment must be adjusted properly, and must stay in the converter’s optimum working range, and proper headroom should be left in order to be prepared for any clips resulting from quantization errors². When we want to represent our signal in the analog domain (i.e. speakers) again, that is called Digital-to-Analog conversion.
FX (Effect) Devices: These include -but not limited to- dynamic processors, equalizers and time-based effects such as reverb, delay, chorus, etc. These processors may be used anywhere in the signal chain between the preamp and the power amp, in the form of an analog equipment, a digital signal processor (DSP) or a computer software (DAW plug-in). Each FX device may have its own input gain, output level, or in the case of digital processors, its own A/D & D/A conversion. Each device also has its own challenges regarding proper gain adjustments, for example tube devices may require high enough input to saturate the tubes, or digital processors with mediocre converters need an input high enough that the noise floor doesn’t become an issue.
Summing: Whenever we want more than one signal to be combined into a single audio signal, we are talking about summing. Summing can be in the analog domain in an analog mixing console, where it’s called Analog Summing; or it can be digital and take place inside the computer software or a digital mixer, where it’s called Digital Summing. Both approaches has its strengths and weaknesses, and these will be discussed in a forthcoming chapter thoroughly.
Power Amp: When we want our line-level signals to be heard from an output transducer (speaker, headphone), we use a power amplifier to amplify this signal so that it’s able to drive a speaker or a headphone.

Gain Structure Elements

This section will cover the detailed explanations of elements that are closely related to proper gain staging. Understanding of these concepts is critical in order to explain what proper gain staging is.

Bit Depth

The bit depth refers to the theoretical dynamic range or resolution of digital audio as it’s quantized to the specified number of bits during the sampling process.

Bit depth directly affects a system’s signal-to-noise ratio, and it’s usable dynamic range. Higher bit rates provide more resolution (wider dynamic range) and lower noise floors. It’s generally accepted that “for each 1-bit increase in bit depth, the SNR will increase by 6 dB”³. This gives us a theoretical dynamic range of 96 dB for 16 bit systems and 144 dB for 24 bit systems.

Dynamic Range

Dynamic range is the difference between the largest and smallest signal a system can record or reproduce; where the lower end is often called the “noise floor”, and the higher end “peak level”.

In analog systems, as dynamic range is rather limited (i.e. tape recorders peaking at 90 dB and radio broadcasts around 50 dB), gain staging plays a crucial role in keeping the noise floor down.

Generally speaking, the higher the dynamic range (thus lower the noise floor), the less susceptible the system is to inadequate signal levels and improper gain staging, as any apparent dynamic range and noise floor beyond 120 db is outside the hearing range of the average human ear⁴.

Another application that can increase the perceived dynamic range is dithering. Renowned engineer Bob Katz described dithering as:

“In an analog system, the signal is continuous, but in a PCM digital system, the amplitude of the signal out of the digital system is limited to one of a set of fixed values or numbers. This process is called quantization. Each coded value is a discrete step… if a signal is quantized without using dither, there will be quantization distortion related to the original input signal… In order to prevent this, the signal is “dithered”, a process that mathematically removes the harmonics or other highly undesirable distortions entirely, and that replaces it with a constant, fixed noise level.”⁵

Headroom

Figure 2: Example of a Dynamic Range

Referring to the figure above, we can define headroom as the safety zone between a given signal and the peak level of a given system be it analog or digital (See Figure 2)⁶.

Leaving enough headroom during any stage of gain structuring is important, as it allows us to deal with sudden increases in the audio signal effectively, and avoid digital clipping in the converters due to quantization errors.

Benefits of Proper Gain Structure

Proper gain structuring in any stage of the signal chain has the following apparent benefits:

Increase in perceived dynamic range,
Decrease in perceived noise floor.,
Better quality A/D & D/A conversion by avoiding digital clipping and utilizing the full dynamic range,
Overall Increase in signal quality as the signal stays in the optimum working range of equipments,
Better dealing with analog distortion and digital clipping.

Problems of an Improper Gain Structure

Failure to achieve proper gain structure causes many problems in all stages off the signal chain. These include:

Decrease in perceived dynamic range,
Elevated and problematic noise issues,
Digital clipping due to quantization errors on the higher end of the dynamic range,
Overloaded or underutilized equipment,
Signal more prone to digital clipping or analog distortion.

Metering & Calibration

Knowing how to read signal meters is essential in order to judge the gain adjustments correctly. There are different types of meters and measurements which shows different aspects of a given signal. Understanding these differences allows us to base our gain staging on accurate terms.

Definition of Concepts

Signal Measurements

We measure a given signal’s loudness using Sound Pressure Level (SPL), in decibels, which “is a logarithmic measure of the effective sound pressure of a sound relative to a reference value.”⁷. The common reference level is 20 µPa RMS, which is considered as the threshold of human hearing⁸.

We often use two different decibel scales, dBu (Decibels Unloaded) and dBFS (Decibels Full Scale):

“The dBu scale measures the analog voltage level in our equipment and cabling, with 0 dBu calibrated to about 0.775 volts.

dBFS indicate signal levels as defined by the values of the digital samples of an analog signal that as been converted to digital data. The top of the scale (0 dBFS) indicates a digital value where all the bits of a digital sample have a value of 1.”⁹

Types of Metering

VU (Volume Unit) Meters: These are traditional meters with a slow response (~300 ms), which shows the average loudness of the given signal instead of its sudden peaks and dips. These are calibrated from -20 to +3 dB, where 0 dB VU = Line Level (usually +4 dBu).
PPM (Peak Program Metering) Meters: This type of meters have a faster response time (~10 ms) to show the transient peaks of an incoming signals. These are typically calibrated from -12 to +12 (+14,+18) dB, where 0 dB PPM = 0 dBu (However this 0 dB PPM may be calibrated to any desired dBu value).
dBFS Meters: These meters are digital meters either hardware or software, which represent binary values of a converted signal, and can either show peaks or average signal levels. These do not represent analog voltages or signal levels, thus are not calibrated according to these. The actual calibration occurs inside the ADC circuit, where -18 dBFS often equals +4 dBu.

Reading & Calibrating Meters

Each of the aforementioned meters represent audio signal level differently, and each needs a different approach when deciding where in the meter scale our signal should reside (See Figure 3)⁹.

As mentioned before most professional audio equipment have an optimum working range regarding signal level, and this is called the “Line Level”. It’s safe to assume that in order to achieve the benefits of proper gain staging, we should aim for our signal to be at around this line level, and make our gain adjustments and meter readings accordingly.

Nominal Line level is accepted as +4 dBu in most professional audio equipment¹⁰. Assuming the meters are calibrated as mentioned (standard accepted calibration), the optimum level for any given signal can be summarized as:

VU (0 VU = +4 dBu): As VU meters show the average level, the signal should be around the 0 VU mark most of the time.
PPM (0 PPM = 0 dBu): PPM meters are more responsive to transient peaks, so we should observe peak values closely, leave enough headroom so that the signal stays away from the distortion zone at all times, and aim for around 0 to +4 dB PPM for most of the signal, with occasional peaks allowed for up to +8 or +12 dB PPM. Any higher level may be risky, prone to distortion and doesn’t leave enough headroom.
dBFS (-18 dBFS = +4 dBu at ADC): At the converter stage, any signal going over 0 dBFS will be digitally impossible to represent and will be clipped. Therefore we should allow enough headroom so any occasional peak can be dealt with by the ADC without going over 0 dBFS. Assuming the digital meter is in peak mode; for a 24 bit converter, average signal level around -18 dBFS with transient peaks up to -12 dBFS can be a good compromise between optimal signal level and noise floor. Once the signal is in the digital domain, things start to change. For example, 32 bit or 64 bit floating point internal mixing or processing engines of DAWs or plug-ins allow the digital signals go over 0 dBFS without clipping. This will be described in the next chapter.

Figure 3: Types of Metering Systems

Mixing and Summing Environments

After familiarizing with the important concepts, it’s time to look at gain structuring in real world usage, on specific equipments. As analog and digital systems show differences and similarities on all stages of gain structuring, these systems will be investigated separately, with a detailed comparison summarizing these differences and similarities will be given in a following section.

Gain Staging in an Analog Setup

Analog systems are inherently more susceptible to improper gain staging, due to the lower dynamic range they offer, proneness to noise and generation loss (signals being carried between equipments in the analog domain [cables, tape systems etc.] lose their quality in every step).

That being said, analog systems are generally more forgiving when it comes to signal overload, as the headroom above the nominal line level is tend to be left higher (though noise level is higher, too) and analog distortion is less irritating or even desirable in some cases (tape compression, tube saturation, etc.)

Next few sections will investigate some of the major components in an analog setup in detail, regarding gain structuring, and provide ways to achieve optimum levels and highest signal quality.

Preamps

Whether built into a recording console, or just a standalone unit, preamps are the beginning of the gain structure chain where our low voltage signals are converted to nominal line level (+4 dBu) in order to feed the succeeding equipment with proper levels.

Figure 4: SSL X-Rack VHD Preamp

Common controls and indicators on preamps include:

Gain Controls: These are the controls where we adjust the amplification of our incoming signal. As this gain control determines how much above the noise floor our initial signal will be, how wide our dynamic signal will get, and how much of a headroom will be left, it’s safe to say that this is the single most important control in the whole gain structure chain.

Different manufacturers label the gain control differently (gain, input gain, trim, drive, etc.), or some units have 2-stage gain controls (a coarse gain adjustment, and a fine trimming adjustment). Some tube units have a separate “drive” knob for adjusting the tube saturation in addition to the normal gain control, and some higher end solid-state units have harmonic drive circuits like the SSL’s X-Logic preamp with Variable Harmonic Drive¹¹ (See Figure 4).

Although this “input gain” control approach is sufficient for sculpting the sound with enough amplification, drive and headroom, without an adjustable “output gain” control, we risk of running the output of the preamp too hot for the next equipment in the signal chain. That’s why more sophisticated units have a dedicated output gain control, which allows us to drive the internal preamp circuit as hot as we want (if the sound we’re searching for requires it) and not worry about the fact that output level of the unit may overload the next equipment in the chain. The “output gain” then becomes the most important control as it allows us to keep our signal level as close to nominal line level as possible.
Pad: Some signal sources may be too hot for a preamp to handle, even with the lowest gain settings. That’s why most preamps offer a pad switch to attenuate the incoming signal (-5, -10, -20, or adjustable dBs depending on the unit) so that we can keep the preamp from distorting and utilize the gain control for sculpting the sound.
Meters & Indicators: Most preamps offer some kind of indicator to judge the incoming signal level correctly. These may be in the form of a simple no-frills “clip” light (for indicating that the signal is too hot and may be distorted), or more sophisticated VU and PPM metering for both input and output levels.

These allow us to visually observe the signal level for transients or overloads, and make our gain adjustments more precise. By using the general metering guidelines discussed in the 3rd chapter (keeping the output signal level around the nominal line level), we can visually ensure that our signal stays in the optimum working range.

Some preamps offer no indication of signal level (such as the aforementioned SSL XLogic VHD Pre), and depend on the input level metering of the next equipment in the signal chain (meter-bridge on a mixing console, or level meters of an A/D converter) for visual indication.

Analog Mixing Consoles

Mixers are devices that “mix” two or more sources and sum their outputs into a master bus in a specified number of channels (Usually 2-channels [stereo] in music production).

As most professional audio equipments, mixers operate at line level, thus proper gain staging is essential in order to stay in the mixer’s optimum working range.

Figure 5: Mackie CFX16 MkII

Mixers consists of several gain stages and in order to keep our signal level at the highest quality possible, we should ensure that they are properly adjusted. Looking at these stages one-by-one:

The Channel Strip: This is where our signal enters the mixer, adjusted and routed to a bus and/or sent to an auxiliary send. The input of a channel strip may include a microphone preamp, or it may accept line level sources only. Regardless of the input source, proper gain structuring procedure in the channel input stays the same:
- Set the input correctly, turn gain control to zero,
- Keep all channel strip processes bypassed or flat (EQ, Dynamics, Inserts, etc.),
- Keep channel fader at unity (0 mark),
- Ensure that we’re metering the correct channel (solo the channel if the mixer don’t have separate channel meters.),
- While signal is present, adjust the gain control so that the meter stays around 0 VU, or +4 PPM (nominal line level)
Channel Insert: While there’s often no gain control for external inserts on a mixing console, it’s advised that general proper gain staging principles discussed before are followed on the external processors in the insert path, so that the signal quality is not compromised.
Auxiliary (Aux) Sends: Aux sends allow us to split the signal at the channel strip and send them to a separate auxiliary bus. These auxiliary buses can be feeding headphone amps, or external effect units. While the aux master level controls the output level going into these external devices, we should still make sure the signal level is adequate in the relevant aux send, considering multiple signals from different channels may be feeding the same aux, thus effectively making the channel “aux send” an “input gain” control for that auxiliary bus.
Auxiliary (Aux) Returns: Aux Returns are basically channel strips, often with limited functionality, that are reserved for returns from external outboard equipment, which are fed from aux sends. Thus, general principles discussed in the channel strip section also apply here.
Mix Buses (Main / Sub): This is where the signals from different channel strips are summed and mixed, as groups or as main mix. This is the output of the summing engine. As we start to mix more than one signal, the cumulative level of summed signals will start increase. It’s good practice to keep the main mix also around nominal line level, or 0 VU; in order to do that, we use the channel faders to alter individual signal levels (think channel faders as input gain controls for the summing engine), and we use the master fader as the mixer’s output level control. Although 0 VU is an acceptable level to aim for the main mix (also for most of the signal chain), as mixing console circuits and summing engines differ substantially, driving the summing engine hotter or cooler than 0 VU may have more desirable effects because of the analog circuit characteristics. Higher-end consoles from Neve, SSL or Harrison for example deal with hot signals superbly, and their desirable drive characteristics made a lot of engineers mix hotter than the recommended 0 VU¹².

Tape Recorders

Analog tape recorders are 2-track or multitrack recording devices that record incoming signals into a magnetic medium (tape). Although their usage is substantially diminished in the modern music production world, their unique compression and sound coloring characteristics still enable them to find a place on many engineers’ studios (See Figure 6)¹³. Even software emulations of classic tape recorders are among the most selling plug-ins for various computer systems¹⁴.

Figure 6: Studer A820 Tape Machine

In gain structuring point of view, at the input stage, tape recorders behave like most professional audio equipment and expect, and optimally work with signals at line level (+4 dBu). Channel input meters are visual indicators of the incoming signal level.

However most tape recorders also include a recording level, and this level needs to be calibrated by a professional according to the tape formulation used and level of tape saturation desired. Once calibrated, any source signal, with tape’s input gain set properly, will have the desired amount of tape saturation and good dynamic range with sufficient headroom and minimal noise.

Effect (FX) Units

As mentioned earlier, FX units may be used anywhere in the signal chain between the preamp and the power amp stages, and here too, gain structuring principles are to be followed in order to keep the signal quality as high as possible.

The rule of thumb here is to keep the signal chain as short as possible, to avoid generation loss and signal degradation between different equipment. Keeping the signal at line level going from one device to another ensures that we are keeping the noise level at minimum, while still achieving sufficient headroom.

Most outboard equipment will have input and output level controls, and this helps a great deal while adjusting the signal level to stay around line level. For units without an input gain control, ensuring that the preceding unit’s output level is at line level; and conversely for units without an output level control, ensuring that the succeeding equipment’s input gain is set at line level avoids signal degradation.

Equalizers often change the overall loudness of the incoming signal, thus it’s advised that their output level is set so that the signal stays at nominal level.

Compressors and other dynamic processors change the level of the incoming signal drastically as such the output level or “make-up gain” control should be set that the output signal level stays close to the input signal.

Some tube units or units with internal drive circuits nearly always have output level controls. So we can still overdrive internal circuit to get the desired effect we want, while making sure that our signal stays at line level at the output.

Gain Staging in a Digital Setup

First there was the PCM coding, then the digital tape recorders. Then came computers, hard disk recorders, digital audio workstations, plugins, digital mixers. Today, we live in a world where no music production is completed without entering the digital domain in one way or another.

In a gain staging perspective, digital systems differ significantly from analog systems. First and foremost, there is no generation loss in digital audio, as all that is carried between equipment is mathematical representations of the actual analog signal. Also digital systems offer superior dynamic range in the conversion stage (usually 24 bits - 144 dB theoretical) and practically infinite dynamic range in the processing and summing stages (32+ bits floating point, over 1,500 dB).

Next few sections will investigate some of the major components in a digital setup in detail, regarding gain structuring, and provide ways to achieve optimum levels and highest signal quality.

Analog to Digital Converters

Analog to digital converters (ADC) are the entry point of our signal into the digital domain. As digital conversion is still a tedious process often requires dealing with errors, levels etc., quality converter design is fundamental in keeping the signal quality as high as possible in the digital world.

As digital metering (dBFS) is just a representation of signals in mathematical values, there is no actual calibration against a voltage or dBu level in the digital domain. Instead the actual calibration is made at the input stage of the conversion. While there’s no accepted standard in calibrating the signal level at the converter input into a value in the dB Full Scale, we will assume the most used calibration level of +4 dBu = -18 dBFS throughout this chapter. This level is used in some higher-quality converters, and is one of the user selectable values in others (See Figure 3).

So, line level signals at +4 dBu coming into the input of the converter, will translate to -18 dBFS at its output, which gives us not just a sufficiently high level above the noise floor, but also enough headroom to deal with any peaks or sudden transients in the signal.

The trick here is to get acquainted with the specifications of the converter thoroughly, as there’s no accepted standard among manufacturers regarding voltage levels and input calibration.

Digital Audio Workstations (DAW) and Digital Mixers

This is where things get interesting. Once our signal is in the digital domain, we are usually doing our processing, mixing and summing inside a computer software, or inside a digital mixer (as basic operating principles for both are essentially the same).

While still most DAW manufacturers tell us to keep our recorded signals as close to 0 dBFS as possible without clipping, this has no practical benefits in today’s 24 bit converters with exceptionally low noise levels. On the contrary, it’s quite inadvisable since it leaves us little or no headroom.

That being said, most modern DAWs and digital mixers (and even plug-ins) have 32 or 64 bit floating point internal processing and summing engines which make proper gain staging practices seem quite irrelevant, as signals are allowed to go much higher than 0 dBFS without clipping or overloading. Overloading all of your DAW channels while keeping your master fader down so that the output does not clip (and your DAC is not overloaded), will have no effect on the quality of your signals¹⁵. Because the internal summing engine of the DAW is dealing with the overloaded signals easily as it’s operated in 32 (or 64) bits using floating point math.

So, as long as you don’t overload your digital to analog converter (by keeping your mix level under 0 dBFS), you’re pretty much free regarding your individual channel levels. While theoretically seems tempting, this freedom has some disadvantages in the practical sense, mainly in metering and plug-in effect usage.

Metering in overloaded channels is not visually helpful, as most of the time the DAW meters stays in the very high end of the dBFS scale.

There are also signal quality issues once effect plug-ins come into the picture, which brings us to the next section:

Plug-ins and Effects

Most modern plug-ins, not unlike DAWs, have also 32+ bits internal processing engines, which allow them to deal with internal overloads and signals above 0 dBFS. But still, some plug-ins, especially dynamic effects (compressors, etc.) don’t have an input level control, and behave unexpectedly if inserted into a channel with an overloaded signal, rendering some of its controls unusable and producing unwanted artifacts (See Figure 7)¹⁶.

Figure 7: Unusable threshold control, and unwanted compression in a dynamics processor on an overloaded channel

This can be avoided using gain/trim plug-ins before the offending plug-in, or by simply using lower signal levels in the first place. As mentioned before, there’s practically no need to constantly hit 0 dBFS or above in today’s technology.

“Running a Digital mix right to the top of the scale is like running your SSL mix buss where the VU meters are slammed all the way to the right and you are constantly hitting it at +25. No one will get a good sound running the desk like that. You won’t get a good sounding mix in digital either.”¹⁷,

Then there is the issue of analog modeling plug-ins. As modeling technology got sophisticated, modeled plug-ins got sonically indistinguishable from their hardware counterparts¹⁸. That means those plug-ins behave exactly like the hardware they’re modeled after, which includes gain staging, and an optimum working range at nominal line level. For example, while your DAW can handle 12 dB above the 0 dBFS signals easily, when going into a Waves VEQ4, this level translates as +34 dBu (considering the plugin is calibrated at +4 dBu = -18 dBFS like the hardware unit¹⁹) at the plug-in input! For a hardware emulation expecting a line level input, this means overloading the plug-in’s internal circuit and sonically altering the way the plug-in supposed to work. Skip Burrows (2009) from Sunrise Studios summarize this as:

“Plugins use the same reference as real equipment. Never try and drive them to the top of the digital scale. Don’t try and make your mix look like a master. You don’t do that on an analog console, so why do we do it in-the-box?”

One other advantage of practicing good gain staging in a modern DAW environment is that any analog outboard units inserted into the signal flow (as most modern DAWs allow inserting hardware signal processors via an I/O pair of the audio interface used) benefit from the proper levels going out of our converters. This practice ensures that these devices stay in their optimum working range (line level) and our signal quality is not compromised.

Differences & Similarities between Analog & Digital Systems

Up until this point, we looked at the way gain staging works at analog and digital units separately, while drawing attention to any relevant similarities or differences. This section will aggregate and summarize these comparison elements, and enable us to judge any weaknesses or strengths these systems have clearly.

Similarities between the two systems may be summarized as:

If you rule out the digital conversion process, signal flow stays pretty much the same on both systems
Maintaining a sufficient headroom is an important part of the gain staging process in both systems, as it allows to maximize the efficiency of the equipment used.
We still need to deal with noise issues, which may be present in the source signal or caused by any equipment in the signal chain.
Both systems are prone to signal distortion and overloads should be carefully observed.

While we observe these similarities, there are also some fundamental differences in operation of analog and digital systems. These include:

In analog systems we deal with electrical currents and voltage levels, whereas digital audio is just a mere numerical representation of these voltages.
Digital audio does not have generation loss, as signals that are carried are just numbers; but analog audio signals lose some of its quality in every step of the signal chain.
Digital systems offer theoretically superior dynamic range (144 dB for 24 bits), and excellent noise performance compared to traditional analog systems.
Digital conversion is still not a perfected process, prone to errors (quantization errors, jitter, etc.) and converter quality still plays a major part in the final sound.
While analog summing devices are limited with whatever dynamic range the device has to offer, floating point processing allow digital summing to deal with signals in excess of 1,500 dB!

Conclusion

These research shows that, in a pro audio environment, in order to keep our signal quality as high as possible, we should be paying attention to our gain structure in every stage of the signal chain, which in turn can easily be summarized with these two words: “Line Level”.

Nominal Line Level is an industry accepted standard signal level that nearly all professional audio equipment manufacturer produce their equipment to optimally work at (1.23 Volts or +4 dBu, which translates to different metering practices as 0 VU, +4 PPM or -18 dBFS in professionally calibrated systems).

In a real world music production scenario, we usually deal with hybrid systems (as opposed to a purely digital or analog system) which may include -but not limited to- analog preamps, A/D & D/A converters, computer DAW systems, audio interfaces, analog summing devices, analog & digital signal processors, etc. By practicing proper gain staging, and keeping our signals around line level at all times, we can be sure that each of these equipment are fed with the right amount of signal they require, and there’s minimal or no degradation in our signal chain.

As every production system is different, learning the specifications, calibration levels, metering systems and level maps of the system we’re working on -be it digital or analog- bears great importance, as we base our judgements about gain staging according to these specifications.

References

BusinessWire, 2012, The Nielsen Company & Billboard’s 2011 Music Industry Report, [http://www.businesswire.com/news/home/20120105005547/en/Nielsen-Company-Billboard%E2%80%99s-2011-Music-Industry-Report Retrieved 21.08.2012] ↩
Lagerfeldt, H. 2012, Levels in Digital Audio, popmusic.dk, [http://www.popmusic.dk/download/pdf/levels-in-digital-audio.pdf Retrieved 21.08.2012] ↩
Kester, W. 2009, Taking the Mystery out of the Infamous Formula, “SNR = 6.02N + 1.76dB,” and Why You Should Care, Analog Devices, [http://www.analog.com/static/imported-files/tutorials/MT-001.pdf Retrieved 22.10.2012] ↩
Everest, F.A., Pohlmann, K.C. 2009, Master Handbook of Acoustics, 5th Edition, McGraw-Hill, New York, p. 26-27. ↩
Katz, B. 2002, Mastering Audio: The Art and The Science, 1st Edition, Focal Press, New York, p. 49-50. ↩
Tomarakos, J. 2000, The Relationship of Data Word Size to Dynamic Range and Signal Quality in Digital Audio Processing Applications, [http://www.analog.com/en/content/relationship_data_word_size_dynamic_range/fca.html Retrieved 25.10.2012] ↩
Wikipedia, 2012, Line level, [http://en.wikipedia.org/wiki/Line_level Retrieved 15.08.2012] ↩
Everest, F.A., Pohlmann, K.C. 2009, Master Handbook of Acoustics, 5th Edition, McGraw-Hill, New York, p. 23. ↩
Independent Recording Network, 2007, Audio Studio Recording: Metering and Gain Structure, [http://www.independentrecording.net/irn/resources/metergain/index.htm Retrieved 15.04.2012] ↩ ↩²
Davis, G., Jones, R. 1990, The Sound Reinforcement Handbook, 2nd Edition, Hal Leonard, Milwaukee, p. 35. ↩
Solid State Logic, 2012, XLogic Alpha VHD Pre, [http://www.solid-state-logic.com/music/xlogic%20alpha%20vhd%20pre/ Retrieved 27.10.2012] ↩
Burrows, S. 2009, The Reason Most ITB mixes don’t Sound as good as Analog mixes, Gear Slutz, [http://www.gearslutz.com/board/so-much-gear-so-little-time/463010-reason-most-itb-mixes-don-t-sound-good-analog-mixes-restored.html Retrieved 15.11.2011] ↩
Endless Analog, 2012, CLASP Ready Tape Machines, [http://www.endlessanalog.com/machines Retrieved 28.10.2012] ↩
Universal Audio, 2012, Universal Audio Store Top Selling Plugins, [http://www.uaudio.com/store.html Retrieved 28.10.2012] ↩
Lagerfeldt, H. 2012, Levels in Digital Audio, popmusic.dk, [http://www.popmusic.dk/download/pdf/levels-in-digital-audio.pdf Retrieved 21.08.2012] ↩
Lagerfeldt, H. 2012, Levels in Digital Audio, popmusic.dk, [http://www.popmusic.dk/download/pdf/levels-in-digital-audio.pdf Retrieved 21.08.2012] ↩
Burrows, S. 2009, The Reason Most ITB mixes don’t Sound as good as Analog mixes, Gear Slutz, [http://www.gearslutz.com/board/so-much-gear-so-little-time/463010-reason-most-itb-mixes-don-t-sound-good-analog-mixes-restored.html Retrieved 15.11.2011] ↩
Avid, 2011, Heat Listening Challenge, [http://www.avid.com/us/resources/HEAT-Listening-Challenge Retrieved 27.10.2012] ↩
Author Unknown, 2006, Waves V-Series User Manual, p. 2 ↩

Python Wrapper for GLEIF Public API

Fri, 13 Apr 2018 11:20:00 +0000

This is a small thing that I started as a necessity for work: We needed to get LEI (Legal Entity Identifier) registration information for our customers, and using GLEIF’s (Global LEI Foundation) public API to query things with Python was convenient at that time. Thinking of reusability, I structured it as a module, covering most of our use cases. Code is now in github, maybe it helps others that need an easier way to query LEI numbers.

To access, go to https://github.com/emredjan/leipy and download the repository (in fact you only need the file leipy/gleif.py). You will need requests and dateutil libraries, and optionally pandas if you want DataFrame output of results, which should all be included with a standard anaconda installation.

You can use it as:

from leipy import GLEIF

gleif = GLEIF(api_version='v1')

raw_output, results, results_df = gleif.request(
  ['HWUPKR0MPOU8FGXBT394','7ZW8QJWVPR4P1J1KQY45'],
  return_dataframe=True
)

GLEIF API has a limit of 200 LEIs per request, my wrapper handles that for you, you can provide more than 200 LEIs. It returns the raw json output from the API, a results class and optionally a pandas DataFrame.

Raw json (as a list of dicts), which you can further process as you please, looks like this:

[{'LEI': {'$': 'HWUPKR0MPOU8FGXBT394'},
  'Entity': {'LegalName': {'$': 'Apple Inc.'},
   'LegalAddress': {'Line1': {'$': 'C/O C T Corporation System'},
    'Line2': {'$': '818 West 7th Street'},
    'Line3': {'$': 'Suite 930'},
    'City': {'$': 'Los Angeles'},
    'Region': {'$': 'US-CA'},
    'Country': {'$': 'US'},
    'PostalCode': {'$': '90017'}},
   'HeadquartersAddress': {'Line1': {'$': '1 Infinite Loop'},
    'City': {'$': 'Cupertino'},
    'Region': {'$': 'US-CA'},
    'Country': {'$': 'US'},
    'PostalCode': {'$': '95014'}},
   'BusinessRegisterEntityID': {'@register': 'RA000598', '$': 'C0806592'},
   'LegalJurisdiction': {'$': 'US'},
   'LegalForm': {'$': 'INCORPORATED'},
   'EntityStatus': {'$': 'ACTIVE'}},
  'Registration': {'InitialRegistrationDate': {'$': '2012-06-06T15:53:00.000Z'},
   'LastUpdateDate': {'$': '2017-12-12T21:19:00.000Z'},
   'RegistrationStatus': {'$': 'ISSUED'},
   'NextRenewalDate': {'$': '2018-12-13T00:31:00.000Z'},
   'ManagingLOU': {'$': 'EVK05KS7XY1DEII3R011'},
   'ValidationSources': {'$': 'FULLY_CORROBORATED'}}},
 {'LEI': {'$': '7ZW8QJWVPR4P1J1KQY45'},
  'Entity': {'LegalName': {'$': 'Google LLC'},
   'LegalAddress': {'Line1': {'$': 'C/O Corporation Service Company'},
    'Line2': {'$': '251 Little Falls Drive'},
    'City': {'$': 'Wilmington'},
    'Region': {'$': 'US-DE'},
    'Country': {'$': 'US'},
    'PostalCode': {'$': '19808'}},
   'HeadquartersAddress': {'Line1': {'$': '1600 Amphitheatre Parkway'},
    'City': {'$': 'Mountain View'},
    'Region': {'$': 'US-CA'},
    'Country': {'$': 'US'},
    'PostalCode': {'$': '94043'}},
   'BusinessRegisterEntityID': {'@register': 'RA000602', '$': '3582691'},
   'LegalJurisdiction': {'$': 'US'},
   'LegalForm': {'$': 'LIMITED LIABILITY COMPANY'},
   'EntityStatus': {'$': 'ACTIVE'}},
  'Registration': {'InitialRegistrationDate': {'$': '2012-06-06T15:52:00.000Z'},
   'LastUpdateDate': {'$': '2018-03-28T17:00:00.000Z'},
   'RegistrationStatus': {'$': 'ISSUED'},
   'NextRenewalDate': {'$': '2018-08-17T18:10:00.000Z'},
   'ManagingLOU': {'$': 'EVK05KS7XY1DEII3R011'},
   'ValidationSources': {'$': 'FULLY_CORROBORATED'}}}]

The results class, with easily accesible members for most commonly used LEI information can be accessed as this:

>>> print(results.legal_name)
['Apple Inc.', 'Google LLC']

>>> print(results.lei_reg_status)
['ISSUED', 'ISSUED']

>>> print(results.date_last_updated)
[datetime.datetime(2017, 12, 12, 21, 19, tzinfo=tzutc()),
 datetime.datetime(2018, 3, 28, 17, 0, tzinfo=tzutc())]

And if you opt for a DataFrame, the results will be conveniently flattened and ready for further processing for you:

>>>results_df

As I said, this is more suited to our use case, and it’s like a hobby project for me. If you need a more generic solution for python check out pygleif, or you can always go for the GLEIF REST API itself.

Next steps for me is to add a solution for automatically downloading and parsing XML and CSV files from GLEIF, and putting everything in a package for easier use (pip and/or conda).

Until then, enjoy!

After EuroPython 2017

Sat, 05 Aug 2017 20:20:00 +0000

It’s over. For 2017 that is. One magical week in Rimini came to an end with the closing talks of EuroPython 2017 conference. This was my first truly technical no-nonsense conference, and I appreciated the change of scenery after all the business-enterprise conferences with all the buzzwords and PRs disguised as talks. You can be sure I’ll be coming back for 2018, but I’ll also try to attend other ones like PyData, PyCon, and others.

There was a lot to love, and a little bit to snub, but overall it was one of best decisions I took in 2017, to attend EuroPython.

Breaking it down to bits:

The Good

The People: Unpretentious, welcoming, lovely Italian people and pythonistas from other countries. People you actually have something in common other than wearing suits. I enjoyed every bit of conversation I had, and left with lots of new friends. Even though I went alone, it sure didn’t feel like it.
The talks: The talks were organized by tracks and there was a bit of everything python. My heart lies in stuff related to data science, but I listened to a lot of the other tracks, which opened up my perspective. Most of the talks were quite informative and educative, none of the PR stuff you usually find at enterprise conferences (there were some pitches during the lightning talks, but that doesn’t count :). The keynotes were well prepared and well executed. There were also the non-technical talks with contents like diversity, privacy, bias which sometimes gets forgotten among all the technical stuff, but nevertheless as important.
Italy: Ah, lovely Italy. The beaches, the food, the weather. Coming from Amsterdam, it was a complete change of lifestyle for a week, and it was glorious.
Social stuff: The social event was generally nice, it gave us the opportunity to socialize and chat with people whom we couldn’t due to the hectic atmosphere of the conference. That being said, being a fully-inclusive conference, EuroPython had lots of opportunities to socialize during breaks, lunches and workshops.
Sprints: I had never been in a sprint / hackathon before, but I felt confident enough to take my chances this time. Although not as smoothly organized as the main conference, there were still some bright moments and takeaways that will sure to help me in the future.
Goodies: I won’t lie. I love conference swag. T-shirts, fidget spinners, bags, pins.. I love it all. Bonus: O’Reilly had a stand where they sold their books 40% off. I bought four. My bag was crying.

The Bad

Diversity: It’s an issue. There’s no denying, no sugar coating. The number of women and people of color was embarrassingly low, and we all have to do something about this. There was a great talk by Anna Ravenscroft on this subject, where she suggested that we should all make a conscious effort to overcome our biases, it’s not enough just to be aware.
Wifi: Every. Public. Event. Has. Shitty. Wifi. I thought a Python event would be better, but alas..
The heat!: Fortunately the venue was air-conditioned, summer heat in Italy is unbearable at times. Biking to the venue under 35 degrees didn’t help.
Speakers: Although the majority was ok, some speakers had limited public speaking skill, and I lost interest quite quickly after they started. One thinks that they would be prepared and practiced better for such a high-profile event. Though to be fair, that’s how they’ll improve, talking in front of people, over and over.

The Takeaways

Public speaking: Speaking in front of a room full of people is a skill that’s both important and necessary to have, and any tips & tricks on that front are well appreciated. There were moments of brilliance during some talks, but also some very concrete suggestions on improving your public speaking skills.
Open Source: There’s nothing to be afraid when contributing. Start small, be confident, and make that pull request.
Bias: Overcome your own biases, ignore subconscious patterns, and in turn reduce your models’ biases. It goes down the drain amazingly fast if you’re not careful and aware.

There’s a lot to be said. But these are my main impressions of my week in EuroPython. As I said before, I’ll be there in 2018, hope to see you there as well!

Plotting Distributions with matplotlib and scipy

Wed, 19 Jul 2017 19:00:00 +0000

It’s important to plot distributions of variables when doing exploratory analysis. However, there may be times when you want to see the theoretical distribution on a plot, i.e. when you want to see how much your variable deviates from it, or when you want to decide on a distribution function visually.

Let’s take the normal (gaussian) distribution as an example. The probability density function (pdf) is:

$ f(x|\mu,\sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}} $

Basically, if we have a range of $x$’s, a mean ($\mu$) and a standard deviation ($\sigma$), we can pass them onto this formula and get corresponding $y$ values, which we can then plot using the standard matplotlib plot() function:

Let’s setup the scene first:

%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt

plt.style.use('seaborn') # pretty matplotlib plots
plt.rcParams['figure.figsize'] = (12, 8)

Let’s get our x values, determine a mean and a standard deviation, and setup the formula for the normal pdf:

x = np.linspace(-5, 5, 5000)
mu = 0
sigma = 1

y = (1 / (np.sqrt(2 * np.pi * np.power(sigma, 2)))) * \
    (np.power(np.e, -(np.power((x - mu), 2) / (2 * np.power(sigma, 2)))))

Now we can plot these using:

plt.plot(x, y);

Which is fine and dandy, but it gets quite cumbersome to write those formulas from scratch using numpy and scipy functions for every distribution we want. Some are even really hard to implement, take for example the cumulative distribution function (cdf) for the standard normal distribution:

$ \Phi(x)=\frac{1}{\sqrt{2\pi}}\int_{-\infty }^{x}e^{-t^{2}/2}\,{\rm {d}}t $

Fortunately for us, the people at scipy provided nearly every kind of distribution function in the scipy.stats package. Using that, we can achieve the same result as above in a cleaner, less error-prone code. We can even plot the cdf on top of that:

import scipy.stats as ss

x = np.linspace(-5, 5, 5000)
mu = 0
sigma = 1

y_pdf = ss.norm.pdf(x, mu, sigma) # the normal pdf
y_cdf = ss.norm.cdf(x, mu, sigma) # the normal cdf

plt.plot(x, y_pdf, label='pdf')
plt.plot(x, y_cdf, label='cdf')
plt.legend();

For reuse, it may be a good idea to put these into a function:

import scipy.stats as ss

def plot_normal(x_range, mu=0, sigma=1, cdf=False, **kwargs):
    '''
    Plots the normal distribution function for a given x range
    If mu and sigma are not provided, standard normal is plotted
    If cdf=True cumulative distribution is plotted
    Passes any keyword arguments to matplotlib plot function
    '''
    x = x_range
    if cdf:
        y = ss.norm.cdf(x, mu, sigma)
    else:
        y = ss.norm.pdf(x, mu, sigma)
    plt.plot(x, y, **kwargs)

x = np.linspace(-5, 5, 5000)

plot_normal(x)
plot_normal(x, cdf=True)

plot_normal(x, -2, 1, color='red', lw=2, ls='-', alpha=0.5)
plot_normal(x, 2, 1.2, color='blue', lw=2, ls='-', alpha=0.5)
plot_normal(x, 0, 0.8, color='green', lw=2, ls='-', alpha=0.5);

Given this knowledge, we can now define a function for plotting any kind of distribution. The important bit is to be careful about the parameters of the corresponding scipy.stats function (Some distributions require more than a mean and a standard deviation). You can check those parameters on the official docs for scipy.stats.

The exponential distribution:

import scipy.stats as ss

def plot_exponential(x_range, mu=0, sigma=1, cdf=False, **kwargs):
    '''
    Plots the exponential distribution function for a given x range
    If mu and sigma are not provided, standard exponential is plotted
    If cdf=True cumulative distribution is plotted
    Passes any keyword arguments to matplotlib plot function
    '''
    x = x_range
    if cdf:
        y = ss.expon.cdf(x, mu, sigma)
    else:
        y = ss.expon.pdf(x, mu, sigma)
    plt.plot(x, y, **kwargs)

x = np.linspace(0, 5, 5000)

plot_exponential(x, 0, 1, color='red', lw=2, ls='-', alpha=0.5, label='pdf')
plot_exponential(x, 0, 1, cdf=True, color='blue', lw=2, ls='-', alpha=0.5, label='cdf')
plt.legend();

The F distribution:

import scipy.stats as ss

def plot_f(x_range, dfn, dfd, mu=0, sigma=1, cdf=False, **kwargs):
    '''
    Plots the f distribution function for a given x range, dfn and dfd
    If mu and sigma are not provided, standard f is plotted
    If cdf=True cumulative distribution is plotted
    Passes any keyword arguments to matplotlib plot function
    '''
    x = x_range
    if cdf:
        y = ss.f.cdf(x, dfn, dfd, mu, sigma)
    else:
        y = ss.f.pdf(x, dfn, dfd, mu, sigma)
    plt.plot(x, y, **kwargs)

x = np.linspace(0.001, 5, 5000)

plot_f(x, 10, 10, 0, 1, color='red', lw=2, ls='-', alpha=0.5, label='pdf')
plot_f(x, 10, 10, 0, 1, cdf=True, color='blue', lw=2, ls='-', alpha=0.5, label='cdf')
plt.legend();

The beta distribution:

import scipy.stats as ss

def plot_beta(x_range, a, b, mu=0, sigma=1, cdf=False, **kwargs):
    '''
    Plots the f distribution function for a given x range, a and b
    If mu and sigma are not provided, standard beta is plotted
    If cdf=True cumulative distribution is plotted
    Passes any keyword arguments to matplotlib plot function
    '''
    x = x_range
    if cdf:
        y = ss.beta.cdf(x, a, b, mu, sigma)
    else:
        y = ss.beta.pdf(x, a, b, mu, sigma)
    plt.plot(x, y, **kwargs)

x = np.linspace(0, 1, 5000)

plot_beta(x, 5, 2, 0, 1, color='red', lw=2, ls='-', alpha=0.5, label='pdf')
plot_beta(x, 5, 2, 0, 1, cdf=True, color='blue', lw=2, ls='-', alpha=0.5, label='cdf')
plt.legend();

As you see, we can extend these as far as we like. These theoretical distributions are important to assess visually and get yourself familiarized with. It’s a powerful tool in a data scientist’s belt to determine the distribution of any variable just by looking at its histogram or KDE.

Let me know what you think, hit me up at @emredjan.

Emulating R plots in Python

Tue, 11 Jul 2017 11:20:00 +0000

Update: Cook’s distance lines on last plot, and cleaned up the code a bit!

Recently, as a part of my Summer of Data Science 2017 challenge, I took up the task of reading Introduction to Statistical Learning cover-to-cover, including all labs and exercises, and converting the R labs and exercises into Python. While I’m still at early chapters, I’ve learned a lot already. Some commands can be straightforward replicated in Python, some are surprisingly hard to find equivalents without using custom functions etc. (not that it’s needed to have “exact” equivalents, as python also has powerful features unique to it, but I’m thinking this as more of a learning opportunity).

One of the simplest R commands that doesn’t have a direct equivalent in Python is plot() for linear regression models (wraps plot.lm() when fed linear models). While python has a vast array of plotting libraries, the more hands-on approach of it necessitates some intervention to replicate R’s plot(), which creates a group of diagnostic plots (residual, qq, scale-location, leverage) to assess model performance when applied to a fitted linear regression model.

Let’s see the example in R with the Auto dataset:

model = lm(mpg ~ . - name, data=Auto)

par(mfrow=c(2,2)) # Plot 4 plots in same screen
plot(model)

The interpretation of these plots are a subject on its own, which I won’t delve into here. You can find several resources, if you want. For now, I’ll dive into the Python code.

Let’s start with the necessary imports and setup commands:

%matplotlib inline

import numpy as np
import pandas as pd

import seaborn as sns
import matplotlib.pyplot as plt

import statsmodels.formula.api as smf
from statsmodels.graphics.gofplots import ProbPlot

plt.style.use('seaborn') # pretty matplotlib plots
plt.rc('font', size=14)
plt.rc('figure', titlesize=18)
plt.rc('axes', labelsize=15)
plt.rc('axes', titlesize=18)

Loading the data, and getting rid of NAs:

auto = pd.read_csv('Auto.csv', na_values=['?'])
auto.dropna(inplace=True)
auto.reset_index(drop=True, inplace=True)

The fitted linear regression model, using statsmodels R style formula API:

model_f = 'mpg ~ cylinders + \
                 displacement + \
                 horsepower + \
                 weight + \
                 acceleration + \
                 year + \
                 origin'

model = smf.ols(formula=model_f, data=auto)

model_fit = model.fit()

Calculations required for some of the plots:

# fitted values (need a constant term for intercept)
model_fitted_y = model_fit.fittedvalues

# model residuals
model_residuals = model_fit.resid

# normalized residuals
model_norm_residuals = model_fit.get_influence().resid_studentized_internal

# absolute squared normalized residuals
model_norm_residuals_abs_sqrt = np.sqrt(np.abs(model_norm_residuals))

# absolute residuals
model_abs_resid = np.abs(model_residuals)

# leverage, from statsmodels internals
model_leverage = model_fit.get_influence().hat_matrix_diag

# cook's distance, from statsmodels internals
model_cooks = model_fit.get_influence().cooks_distance[0]

And now, the actual plots:

1. Residual plot

First plot that’s generated by plot() in R is the residual plot, which draws a scatterplot of fitted values against residuals, with a “locally weighted scatterplot smoothing (lowess)” regression line showing any apparent trend.

This one can be easily plotted using seaborn residplot with fitted values as x parameter, and the dependent variable as y. lowess=True makes sure the lowess regression line is drawn. Additional parameters are passed to underlying matplotlib scatter and line functions using scatter_kws and line_kws, also titles and labels are set using matplotlib methods. The ; in the end gets rid of the output text <matplotlib.text.Text at 0x000000000> at the top of the plot ¹. Top 3 absolute residuals are also annotated:

plot_lm_1 = plt.figure(1)
plot_lm_1.set_figheight(8)
plot_lm_1.set_figwidth(12)

plot_lm_1.axes[0] = sns.residplot(model_fitted_y, 'mpg', data=auto,
                                  lowess=True,
                                  scatter_kws={'alpha': 0.5},
                                  line_kws={'color': 'red', 'lw': 1, 'alpha': 0.8})

plot_lm_1.axes[0].set_title('Residuals vs Fitted')
plot_lm_1.axes[0].set_xlabel('Fitted values')
plot_lm_1.axes[0].set_ylabel('Residuals')


# annotations
abs_resid = model_abs_resid.sort_values(ascending=False)
abs_resid_top_3 = abs_resid[:3]

for i in abs_resid_top_3.index:
    plot_lm_1.axes[0].annotate(i, 
                               xy=(model_fitted_y[i], 
                                   model_residuals[i]));

2. QQ plot

This one shows how well the distribution of residuals fit the normal distribution. This plots the standardized (z-score) residuals against the theoretical normal quantiles. Anything quite off the diagonal lines may be a concern for further investigation.

For this, I’m using ProbPlot and its qqplot method from statsmodels graphics API. statsmodels actually has a qqplot method that we can use directly, but it’s not very customizable, hence this two-step approach. Annotations were a bit tricky, as theoretical quantiles from ProbPlot are already sorted:

QQ = ProbPlot(model_norm_residuals)
plot_lm_2 = QQ.qqplot(line='45', alpha=0.5, color='#4C72B0', lw=1)

plot_lm_2.set_figheight(8)
plot_lm_2.set_figwidth(12)

plot_lm_2.axes[0].set_title('Normal Q-Q')
plot_lm_2.axes[0].set_xlabel('Theoretical Quantiles')
plot_lm_2.axes[0].set_ylabel('Standardized Residuals');

# annotations
abs_norm_resid = np.flip(np.argsort(np.abs(model_norm_residuals)), 0)
abs_norm_resid_top_3 = abs_norm_resid[:3]

for r, i in enumerate(abs_norm_resid_top_3):
    plot_lm_2.axes[0].annotate(i, 
                               xy=(np.flip(QQ.theoretical_quantiles, 0)[r],
                                   model_norm_residuals[i]));

3. Scale-Location Plot

This is another residual plot, showing their spread, which you can use to assess heteroscedasticity.

It’s essentially a scatter plot of absolute square-rooted normalized residuals and fitted values, with a lowess regression line. Scatterplot is a standard matplotlib function, lowess line comes from seaborn regplot. Top 3 absolute square-rooted normalized residuals are also annotated:

plot_lm_3 = plt.figure(3)
plot_lm_3.set_figheight(8)
plot_lm_3.set_figwidth(12)

plt.scatter(model_fitted_y, model_norm_residuals_abs_sqrt, alpha=0.5)
sns.regplot(model_fitted_y, model_norm_residuals_abs_sqrt, 
            scatter=False, 
            ci=False, 
            lowess=True,
            line_kws={'color': 'red', 'lw': 1, 'alpha': 0.8})

plot_lm_3.axes[0].set_title('Scale-Location')
plot_lm_3.axes[0].set_xlabel('Fitted values')
plot_lm_3.axes[0].set_ylabel('$\sqrt{|Standardized Residuals|}$');

# annotations
abs_sq_norm_resid = np.flip(np.argsort(model_norm_residuals_abs_sqrt), 0)
abs_sq_norm_resid_top_3 = abs_sq_norm_resid[:3]

for i in abs_norm_resid_top_3:
    plot_lm_3.axes[0].annotate(i, 
                               xy=(model_fitted_y[i], 
                                   model_norm_residuals_abs_sqrt[i]));

4. Leverage plot

This plot shows if any outliers have influence over the regression fit. Anything outside the group and outside “Cook’s Distance” lines, may have an influential effect on model fit.

statsmodels has a built-in leverage plot for linear regression, but again, it’s not very customizable. Digging around the source of the statsmodels.graphics package, it’s pretty straightforward to implement it from scratch and customize with standard matplotlib functions. There are three parts to this plot: First is the scatterplot of leverage values (got from statsmodels fitted model using get_influence().hat_matrix_diag) vs. standardized residuals. Second one is the lowess regression line for that. And the third and the most tricky part is the Cook’s distance lines, which I currently couldn’t figure out how to draw in Python. But statsmodels has Cook’s distance already calculated, so we can use that to annotate top 3 influencers on the plot:

Update: I think I figured out how to draw Cook’s distance ($D_i$) contours for $D_i=0.5$ and $D_i=1$ The trick was rearranging the formula $p{D_i} = r_i^2 h_i/(1-h_i)$ to plot the lines at 0.5 and 1.

plot_lm_4 = plt.figure(4)
plot_lm_4.set_figheight(8)
plot_lm_4.set_figwidth(12)

plt.scatter(model_leverage, model_norm_residuals, alpha=0.5)
sns.regplot(model_leverage, model_norm_residuals, 
            scatter=False, 
            ci=False, 
            lowess=True,
            line_kws={'color': 'red', 'lw': 1, 'alpha': 0.8})

plot_lm_4.axes[0].set_xlim(0, 0.20)
plot_lm_4.axes[0].set_ylim(-3, 5)
plot_lm_4.axes[0].set_title('Residuals vs Leverage')
plot_lm_4.axes[0].set_xlabel('Leverage')
plot_lm_4.axes[0].set_ylabel('Standardized Residuals')

# annotations
leverage_top_3 = np.flip(np.argsort(model_cooks), 0)[:3]

for i in leverage_top_3:
    plot_lm_4.axes[0].annotate(i, 
                               xy=(model_leverage[i], 
                                   model_norm_residuals[i]))
    
# shenanigans for cook's distance contours
def graph(formula, x_range, label=None):
    x = x_range
    y = formula(x)
    plt.plot(x, y, label=label, lw=1, ls='--', color='red')

p = len(model_fit.params) # number of model parameters

graph(lambda x: np.sqrt((0.5 * p * (1 - x)) / x), 
      np.linspace(0.001, 0.200, 50), 
      'Cook\'s distance') # 0.5 line

graph(lambda x: np.sqrt((1 * p * (1 - x)) / x), 
      np.linspace(0.001, 0.200, 50)) # 1 line

plt.legend(loc='upper right');

I learned a lot, and continuing to learn during this code porting. It’ll hopefully be helpful for others struggling with similar issues.

Cheers.

Validating XML using lxml in Python

Sat, 08 Apr 2017 16:20:00 +0000

Often when working with XML documents, it’s required that we validate our document with a predefined schema. These schemas usually come in XSD (XML Schema Definition) files and while there are commercial and open source applications that can do these validations, it’s more flexible and a good learning experience to do it using Python.

Prerequisites

You need Python installed obviously (I’ll be using Python 3, but the codes should work in Python 2 with minimal modifications). You’ll also need the lxml package to handle schema validations. You can install it using pip:

$ pip install lxml

Or if you’re a conda user:

$ conda install lxml

Importing and using lxml

For XML schema validation, we need the etree module from the lxml package. Let’s also import StringIO from the io package for passing strings as files to etree, as well as sys for handling input.

from lxml import etree
from io import StringIO
import sys

I prefer giving file names as command line arguments to the python file as it simplifies the handling:

filename_xml = sys.argv[1]
filename_xsd = sys.argv[2]

Let’s open and read both files:

# open and read schema file
with open(filename_xsd, 'r') as schema_file:
    schema_to_check = schema_file.read()

# open and read xml file
with open(filename_xml, 'r') as xml_file:
    xml_to_check = xml_file.read()

Parsing XML and XSD files

We can parse the XML files/schemas using the etree.parse() method, and we can load the schema to memory using etree.XMLSchema(). As schemas usually arrive well-formed and correctly formatted, I skipped error checking here for the schema parsing.

xmlschema_doc = etree.parse(StringIO(schema_to_check))
xmlschema = etree.XMLSchema(xmlschema_doc)

Next is the parsing of the actual XML document. I usually do error checking here to catch syntax errors and not well-formed XML documents. lxml throws and etree.XMLSyntaxError exception if it finds errors in the XML document and provides an error_log in the exception. We can write this to a file check the incorrect lines and tags:

# parse xml
try:
    doc = etree.parse(StringIO(xml_to_check))
    print('XML well formed, syntax ok.')

# check for file IO error
except IOError:
    print('Invalid File')

# check for XML syntax errors
except etree.XMLSyntaxError as err:
    print('XML Syntax Error, see error_syntax.log')
    with open('error_syntax.log', 'w') as error_log_file:
        error_log_file.write(str(err.error_log))
    quit()

except:
    print('Unknown error, exiting.')
    quit()

Validating with Schema

At the final step we can validate our XML document against the XSD schema using assertValid method from etree.XMLSchema. This method will get our parsed XML file (in variable doc above) and try to validate it using the schema definitions. It throws an etree.DocumentInvalid exception with an error_log object as above. We can also write this to a file to check any invalid tags or values.

# validate against schema
try:
    xmlschema.assertValid(doc)
    print('XML valid, schema validation ok.')

except etree.DocumentInvalid as err:
    print('Schema validation error, see error_schema.log')
    with open('error_schema.log', 'w') as error_log_file:
        error_log_file.write(str(err.error_log))
    quit()

except:
    print('Unknown error, exiting.')
    quit()

You can save this script (i.e. as ‘validation.py’) and use it with:

$ python validation.py <path_to_xml_file> <path_to_xsd_file>

Any errors will be written to ‘error_syntax.log’ and ‘error_schema.log’ files (in the same directory as your .py file) with timestamps, line number and detailed explanation of validation errors. You can check and correct your XML documents before validating using this script again.

lxml is quite an extensive and flexible package to handle and process XML and related files. Check the sources below for tutorials, references and more information.

Sources

Upgrade bash in Windows 10

Thu, 06 Apr 2017 20:20:00 +0000

One of the not-much-advertised featues of the new Creators update for Windows 10 is the support for Ubuntu 16.04 in Windows Subsytem for Linux (WSL). The catch is, if you installed the WSL prior to upgrading Windows, it will not upgrade it for you, and leave it as Ubuntu 14.04.

If you want the latest LTS Ubuntu, one option is delete the WSL and reinstall using (in powershell):

lxrun /uninstall
lxrun /install

This, however will delete all your linux settings / files under WSL, which may be an inconvenience. A less destructive option is to just use (in bash):

sudo do-release-upgrade

Which works surprisingly well, and doesn’t take too long to complete on a modern system. See for yourself:

Before upgrade:

Upgrading:

After upgrade:

Bonus: libinotify now works and so does jekyll --watch (more info)

Commiting is Fun

Fri, 17 Mar 2017 19:20:00 +0000

I always shy away from contributing to projects in GitHub because:

I’m still a newbie when it comes to coding,
I still have the fear of failure, although my MBA is helping immensely in that regard

Having said that, actually making an effort to propose solutions to problems I face in programs I use every day, is a rewarding experience. To see your changes working, to see that pull request getting accepted gives me a sense of satisfaction I don’t usually get anywhere else. I probably sound like a junior CS student, but bear with me, in some ways, I am.

My current contributions are limited to small add-ons for programs I use and my own testing repositories. But as I get the hang of it and contribute more, I gain confidence and that motivates me to code more. Now, that’s a cycle I like.

I also started building my Stack Overflow profile. I may be new to development, but I have a lot of practical experience, I’m a quick learner, and I love researching stuff. I might as well help the community in ways I can.

Oh, and this and this motivated me to write this after a long break. See ya.

A New Chapter

Sun, 04 Sep 2016 19:21:00 +0000

It’s the opening week! My MBA starts on Tuesday with some refresher courses on databases / SQL and statistics / econometrics. I’m scared but really excited to be studying again. It will be a tough two years that’s for sure, with my work and evening classes, but hopefully I’ll manage.

Meanwhile I’ll try to get busy with the new Kaggle dataset on house prices. I’ll be sure to share my findings / insights here. Until then, happy school year to you all :)

First Post

Tue, 12 Jul 2016 12:31:19 +0000

This is here for the sole purpose of reminding me that the site went live on this date. I hope this will remind me to get off my ass and update the blog more often.