Robert Edmonds' bloghttp://blog.mycre.ws/2016-08-06T21:52:00+00:00Cable modems: Arris SB6190 vs. Netgear CM6002016-08-06T21:52:00+00:00Robert Edmondstag:blog.mycre.ws,2016-08-06:articles/cable-modems-arris-sb6190-vs-netgear-cm600/<p>Recently I activated new cable ISP service at my home and needed to purchase a
cable modem. There were only a few candidate devices that supported at least 24
downstream channels (preferably 32), and did not contain an integrated router or
access point.</p>
<p>The first modem I tried was the <a href="http://mydeviceinfo.xfinity.com/device.php?devid=461">Arris SB6190</a>, which supports 32 downstream
channels. It is based on the <a href="http://www.intel.com/content/www/us/en/connected-home/ibc-puma-6-32-channel-tech-brief.html">Intel Puma 6</a> SoC, and looking at an older release
of the <a href="https://sourceforge.net/projects/sb6190.arris/files/">SB6190 firmware source</a> reveals that the device runs Linux. This device,
running the latest 9.1.93N firmware, goes into a failure mode after several days
of uptime which causes it to drop 1-2% of packets. Here is a <a href="http://oss.oetiker.ch/smokeping/">SmokePing</a> graph
that measures latency to my ISP's recursive DNS server, showing the transition
into the “degraded” mode:</p>
<p><img alt="SmokePing Arris SB6190 Firmware 9.1.93N" src="http://blog.mycre.ws/articles/cable-modems-arris-sb6190-vs-netgear-cm600/images/SmokePing_Arris_SB6190_Firmware_9.1.93N.png" /></p>
<p>It didn't drop packets at random, though. Some traffic would be
deterministically dropped, such as the parallel A/AAAA DNS lookups generated by
the glibc DNS stub resolver. For instance, in the following <code>tcpdump</code> output:</p>
<pre><code class="boxed">[1] 17:31:46.989073 IP [My IP].50775 > 75.75.75.75.53: 53571+ A? www.comcast6.net. (34)
[2] 17:31:46.989102 IP [My IP].50775 > 75.75.75.75.53: 14987+ AAAA? www.comcast6.net. (34)
[3] 17:31:47.020423 IP 75.75.75.75.53 > [My IP].50775: 53571 2/0/0 CNAME comcast6.g.comcast.net., […]
[4] 17:31:51.993680 IP [My IP].50775 > 75.75.75.75.53: 53571+ A? www.comcast6.net. (34)
[5] 17:31:52.025138 IP 75.75.75.75.53 > [My IP].50775: 53571 2/0/0 CNAME comcast6.g.comcast.net., […]
[6] 17:31:52.025282 IP [My IP].50775 > 75.75.75.75.53: 14987+ AAAA? www.comcast6.net. (34)
[7] 17:31:52.056550 IP 75.75.75.75.53 > [My IP].50775: 14987 2/0/0 CNAME comcast6.g.comcast.net., […]
</code></pre>
<p>Packets <code>[1]</code> and <code>[2]</code> are the A and AAAA queries being initiated in parallel.
Note that they both use the same 4-tuple of <em>(Source IP, Destination IP, Source
Port, Destination Port)</em>, but with different DNS IDs. Packet <code>[3]</code> is the
response to packet <code>[1]</code>. The response to packet <code>[2]</code> never arrives, and five
seconds later, the glibc stub resolver times out and retries in <code>single-request</code>
mode, which performs the A and AAAA queries sequentially. Packets <code>[4]</code> and
<code>[5]</code> are the type A query and response, and packets <code>[6]</code> and <code>[7]</code> are the
AAAA query and response.</p>
<p>The Arris SB6190 running firmware 9.1.93N would consistently interfere with
these parallel DNS requests, but only when operating in its “degraded” mode. It
also didn't matter whether glibc was configured to use an IPv4 or IPv6
nameserver, or which nameserver was being used. Power cycling the modem would
fix the issue for a few days.</p>
<p>My ISP offered to downgrade the firmware on the Arris SB6190 to version 9.1.93K.
This firmware version doesn't go into a degraded mode after a few days, but it
does exhibit higher latency, and more jitter:</p>
<p><img alt="SmokePing Arris SB6190 Firmware 9.1.93K" src="http://blog.mycre.ws/articles/cable-modems-arris-sb6190-vs-netgear-cm600/images/SmokePing_Arris_SB6190_Firmware_9.1.93K.png" /></p>
<p>It seemed unlikely that Arris would fix the firmware issues in the SB6190 before
the end of my 30-day return window, so I returned the SB6190 and purchased a
<a href="http://mydeviceinfo.xfinity.com/device.php?devid=455">Netgear CM600</a>. This modem appears to be based on the <a href="http://www.broadcom.com/products/Cable/Cable-Modem-Solutions/BCM3384">Broadcom BCM3384</a> and
looking at an older release of the <a href="http://kb.netgear.com/app/answers/detail/a_id/2649/~/netgear-open-source-code-for-programmers-(gpl)">CM600 firmware source</a> reveals that the
device runs the open source <a href="https://en.wikipedia.org/wiki/ECos">eCos</a> embedded operating system.</p>
<p>The Netgear CM600 so far hasn't exhibited any of the issues I found with the
Arris SB6190 modem. Here is a SmokePing graph for the CM600, which shows median
latency about 1 ms lower than the Arris modem:</p>
<p><img alt="SmokePing Netgear CM600" src="http://blog.mycre.ws/articles/cable-modems-arris-sb6190-vs-netgear-cm600/images/SmokePing_Netgear_CM600.png" /></p>
<p>It's not clear which company is to blame for the problems in the Arris modem.
Looking at the DOCSIS drivers in the SB6190 firmware source reveals copyright
statements from ARRIS Group, Texas Instruments, and Intel. However, I would
recommend avoiding cable modems produced by Arris in particular, and cable
modems based on the Intel Puma SoC in general.</p>Works with Debian: Intel SSD 750, AMD FirePro W4100, Dell P2715Q2015-12-13T08:58:00+00:00Robert Edmondstag:blog.mycre.ws,2015-12-13:articles/works-with-debian-intel-ssd-750-amd-firepro-w4100-dell-p2715q/<p>I recently installed new hardware in my primary computer running Debian
unstable. The disk used for the <code>/</code> and <code>/home</code> filesystem was replaced with an
<a href="http://www.intel.com/content/www/us/en/solid-state-drives/solid-state-drives-750-series.html">Intel SSD 750 series</a> NVM Express card. The graphics card was replaced by an
<a href="http://www.amd.com/en-us/products/graphics/workstation/firepro-3d/4100">AMD FirePro W4100</a> card, and two <a href="http://accessories.us.dell.com/sna/productdetail.aspx?c=us&cs=04&l=en&sku=210-ADOF">Dell P2715Q</a> monitors were installed.</p>
<h3><a href="http://www.intel.com/content/www/us/en/solid-state-drives/solid-state-drives-750-series.html">Intel SSD 750 series</a> NVM Express card</h3>
<p>This is an 800 GB SSD on a PCI-Express x4 card (model number <code>SSDPEDMW800G4X1</code>)
using the relatively new <a href="https://en.wikipedia.org/wiki/NVM_Express">NVM Express</a> interface, which appears as a
<code>/dev/nvme*</code> device. The stretch alpha 4 Debian installer was able to detect and
install onto this device, but <a href="https://tracker.debian.org/pkg/grub-installer">grub-installer</a> 1.127 on the installer media was
unable to install the boot loader. This was due to a bug recently fixed in
1.128:</p>
<pre><code class="boxed">grub-installer (1.128) unstable; urgency=high
* Fix buggy /dev/nvme matching in the case statement to determine
disc_offered_devfs (Closes: #799119). Thanks, Mario Limonciello!
-- Cyril Brulebois <kibi@debian.org> Thu, 03 Dec 2015 00:26:42 +0100
</code></pre>
<p>I was able to download and install the updated .udeb by hand in the installer
environment and complete the installation.</p>
<p>This card was installed on a <a href="http://www.supermicro.com/products/motherboard/Xeon/C220/X10SAE.cfm">Supermicro X10SAE</a> motherboard, and the UEFI BIOS
was able to boot Debian directly from the NVMe card, although I updated to the
latest available BIOS firmware prior to the installation. It appears in <code>lspci</code>
like this:</p>
<pre><code class="boxed">02:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)
(prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Flags: bus master, fast devsel, latency 0
Memory at f7d10000 (64-bit, non-prefetchable) [size=16K]
Expansion ROM at f7d00000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [150] Virtual Channel
Capabilities: [180] Power Budgeting <?>
Capabilities: [190] Alternative Routing-ID Interpretation (ARI)
Capabilities: [270] Device Serial Number 55-cd-2e-41-4c-90-a8-97
Capabilities: [2a0] #19
Kernel driver in use: nvme
</code></pre>
<p>The card itself appears very large in marketing photos, but this is a visual
trick: the photographs are taken with the low-profile PCI bracket installed,
rather than the <a href="https://commons.wikimedia.org/wiki/File:Intel_SSD_750_series,_400_GB_add-in_card_model,_top_view.jpg">standard height PCI bracket</a> which it ships installed with.</p>
<p>smartmontools <a href="http://marc.info/?l=smartmontools-support&m=141555614628846&w=2">fails to read SMART data</a> from the drive, although it is still
able to retrieve basic device information, including the temperature:</p>
<pre><code class="boxed">root@chase{0}:~# smartctl -d scsi -a /dev/nvme0n1
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.3.0-trunk-amd64] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: NVMe
Product: INTEL SSDPEDMW80
Revision: 0135
Compliance: SPC-4
User Capacity: 800,166,076,416 bytes [800 GB]
Logical block size: 512 bytes
Rotation Rate: Solid State Device
Logical Unit id: 8086INTEL SSDPEDMW800G4 1000CVCQ531500K2800EGN
Serial number: CVCQ531500K2800EGN
Device type: disk
Local Time is: Sun Dec 13 01:48:37 2015 EST
SMART support is: Unavailable - device lacks SMART capability.
=== START OF READ SMART DATA SECTION ===
Current Drive Temperature: 31 C
Drive Trip Temperature: 85 C
Error Counter logging not supported
[GLTSD (Global Logging Target Save Disable) set. Enable Save with '-S on']
Device does not support Self Test logging
root@chase{4}:~#
</code></pre>
<p>Simple tests with <code>cat /dev/nvme0n1 >/dev/null</code> and <code>iotop</code> show that the card
can read data at about 1 GB/sec, about twice as fast as the SATA-based SSD that
it replaced. apt/dpkg now run about as fast on the NVMe SSD as they do on a
<code>tmpfs</code>.</p>
<p>Hopefully this device doesn't at some point require updated firmware, like some
<a href="http://www.anandtech.com/show/9196/samsung-releases-second-840-evo-fix">infamous SSDs have</a>.</p>
<h3><a href="http://www.amd.com/en-us/products/graphics/workstation/firepro-3d/4100">AMD FirePro W4100</a> graphics card</h3>
<p>This is a graphics card capable of driving multiple DisplayPort displays at
<a href="https://en.wikipedia.org/wiki/4K_resolution#Ultra_HD">"4K" resolution</a> and at a 60 Hz refresh rate. It has four Mini DisplayPort
connectors, although I only use two of them.</p>
<p>It was difficult to find a sensible graphics card. Most discrete graphics cards
<a href="http://femfreq.tumblr.com/image/84553495875">appear to be marketed towards video gamers</a> who apparently must seek out bulky
cards that occupy multiple PCI slots and have excessive cooling devices. (To
take a random example, the <a href="https://www.asus.com/us/Graphics-Cards/STRIXR9390XDC3OC8GD5GAMING/">ASUS STRIX R9 390X</a> has three fans and brags about
its "Mega Heatpipes".)</p>
<p>AMD markets a separate line of <a href="https://en.wikipedia.org/wiki/AMD_FirePro">"FirePro" graphics cards</a> intended for
professionals rather than gamers, although they appear to be based on the same
GPUs as their "Radeon" video cards. The <a href="http://www.amd.com/en-us/products/graphics/workstation/firepro-3d/4100">AMD FirePro W4100</a> is a normal
half-height PCI-E card that fits into a single PCI slot and has a relatively
small cooler with a single fan. It doesn't even require an auxilliary power
connection and is <a href="https://commons.wikimedia.org/wiki/File:Trident_video_card.png">about the same dimensions</a> as older video cards that I've
successfully used with Debian.</p>
<p>It was difficult to determine whether the W4100 card was actually supported by
an open source driver before buying it. The word "FirePro" appears nowhere on
the webpage for the <a href="http://xorg.freedesktop.org/wiki/RadeonFeature/">X.org Radeon driver</a>, but I was able to find a "CAPE VERDE"
listed as an engineering name which appears to match the "Cape Verde" code name
for the FirePro W4100 given on Wikipedia's <a href="https://en.wikipedia.org/wiki/List_of_AMD_graphics_processing_units#FirePro_Workstation_Series_.28Wx100.29">List of AMD graphics processing
units</a>. This explains the "verde" string that appears in the firmware filenames
requested by the kernel (available only in the <a href="https://tracker.debian.org/pkg/firmware-nonfree">non-free/firmware-amd-graphics
package</a>):</p>
<pre><code class="boxed">[drm] initializing kernel modesetting (VERDE 0x1002:0x682C 0x1002:0x2B1E).
[drm] Loading verde Microcode
radeon 0000:01:00.0: firmware: direct-loading firmware radeon/verde_pfp.bin
radeon 0000:01:00.0: firmware: direct-loading firmware radeon/verde_me.bin
radeon 0000:01:00.0: firmware: direct-loading firmware radeon/verde_ce.bin
radeon 0000:01:00.0: firmware: direct-loading firmware radeon/verde_rlc.bin
radeon 0000:01:00.0: firmware: direct-loading firmware radeon/verde_mc.bin
radeon 0000:01:00.0: firmware: direct-loading firmware radeon/verde_smc.bin
</code></pre>
<p>The card appears in <code>lspci</code> like this:</p>
<pre><code class="boxed">01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde GL [FirePro W4100]
(prog-if 00 [VGA controller])
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 2b1e
Flags: bus master, fast devsel, latency 0, IRQ 55
Memory at e0000000 (64-bit, prefetchable) [size=256M]
Memory at f7e00000 (64-bit, non-prefetchable) [size=256K]
I/O ports at e000 [size=256]
Expansion ROM at f7e40000 [disabled] [size=128K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Capabilities: [58] Express Legacy Endpoint, MSI 00
Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [150] Advanced Error Reporting
Capabilities: [200] #15
Capabilities: [270] #19
Kernel driver in use: radeon
</code></pre>
<p>The W4100 appears to work just fine, except for a few bizarre error messages
that are printed to the kernel log when the displays are woken from power saving
mode:</p>
<pre><code class="boxed">[Sun Dec 13 00:24:41 2015] [drm:si_dpm_set_power_state [radeon]] *ERROR* si_enable_smc_cac failed
[Sun Dec 13 00:24:41 2015] [drm:si_dpm_set_power_state [radeon]] *ERROR* si_enable_smc_cac failed
[Sun Dec 13 00:24:41 2015] [drm:radeon_dp_link_train [radeon]] *ERROR* displayport link status failed
[Sun Dec 13 00:24:41 2015] [drm:radeon_dp_link_train [radeon]] *ERROR* clock recovery failed
[Sun Dec 13 00:24:41 2015] [drm:radeon_dp_link_train [radeon]] *ERROR* displayport link status failed
[Sun Dec 13 00:24:41 2015] [drm:radeon_dp_link_train [radeon]] *ERROR* clock recovery failed
[Sun Dec 13 00:24:41 2015] [drm:si_dpm_set_power_state [radeon]] *ERROR* si_enable_smc_cac failed
[Sun Dec 13 00:24:41 2015] [drm:radeon_dp_link_train [radeon]] *ERROR* displayport link status failed
[Sun Dec 13 00:24:41 2015] [drm:radeon_dp_link_train [radeon]] *ERROR* clock recovery failed
[Sun Dec 13 00:24:41 2015] [drm:radeon_dp_link_train [radeon]] *ERROR* displayport link status failed
[Sun Dec 13 00:24:41 2015] [drm:radeon_dp_link_train [radeon]] *ERROR* clock recovery failed
</code></pre>
<p>There don't appear to be any ill effects from these error messages, though. I
have the following package versions installed:</p>
<pre><code class="boxed">||/ Name Version Description
+++-=============================-===================-================================================
ii firmware-amd-graphics 20151207-1 Binary firmware for AMD/ATI graphics chips
ii linux-image-4.3.0-trunk-amd64 4.3-1~exp2 Linux 4.3 for 64-bit PCs
ii xserver-xorg-video-radeon 1:7.6.1-1 X.Org X server -- AMD/ATI Radeon display driver
</code></pre>
<p>The <a href="http://www.supermicro.com/products/motherboard/Xeon/C220/X10SAE.cfm">Supermicro X10SAE</a> motherboard has two PCI-E 3.0 slots, but they're listed
as functioning in either "16/NA" or "8/8" mode, which apparently means that
putting anything in the second slot (like the Intel 750 SSD, which uses an x4
link) causes the video card to run at a smaller x8 link width. This can be
verified by looking at the widths reported in the "LnkCap" and "LnkSta" lines in
the <code>lspci -vv</code> output:</p>
<pre><code class="boxed">root@chase{0}:~# lspci -vv -s 01:00.0 | egrep '(LnkCap|LnkSta):'
LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
root@chase{0}:~#
</code></pre>
<p>I did not notice any visible artifacts or performance degradation because of
the smaller link width.</p>
<p>The <code>sensors</code> utility from the <code>lm-sensors</code> package is capable of reporting the
temperature of the GPU:</p>
<pre><code class="boxed">root@chase{0}:~# sensors radeon-pci-0100
radeon-pci-0100
Adapter: PCI adapter
temp1: +55.0°C (crit = +120.0°C, hyst = +90.0°C)
root@chase{0}:~#
</code></pre>
<h2><a href="http://accessories.us.dell.com/sna/productdetail.aspx?c=us&cs=04&l=en&sku=210-ADOF">Dell P2715Q</a> monitors</h2>
<p>Two new 27" Dell monitors with a native resolution of 3840x2160 were attached
to the new graphics card. They replaced two ten year old <a href="http://www.dell.com/downloads/global/products/monitors/en/spec_2001fp_en.pdf">Dell 2001FP</a> monitors
with a native resolution of 1600x1200 that had experienced burn-in, providing
4.32 times as many pixels. (TV and monitor manufacturers now shamelessly refer
to the 3840x2160 resolution as <a href="https://en.wikipedia.org/wiki/4K_resolution#Ultra_HD">"4K" resolution</a> even though neither dimension
reaches 4000 pixels.)</p>
<p>There was very little to setup beyond plugging the DisplayPort inputs on these
monitors into the DisplayPort outputs on the graphics card. Most of the setup
involved reconfiguring software to work with the very high resolution.</p>
<p>X.org, for tl;dr <a href="https://bugs.freedesktop.org/show_bug.cgi?id=23705">CLOSED NOTABUG</a> reasons doesn't set the DPI correctly. These
monitors have ~163 DPI resolution, so I added <code>-dpi 168</code> to
<code>/etc/X11/xdm/Xservers</code>. (168 is an even 1.75x multiple of 96.) Software like
Google Chrome and xfce4-terminal rendered fonts and graphical elements at the
right size, but other software like notion, pidgin, and virt-manager did not
fully understand the high DPI. E.g., pidgin renders fonts at the correct size,
but icons are too small.</p>
<p>The default X cursor was also too small. To fix this, I installed the
<a href="https://tracker.debian.org/pkg/dmz-cursor-theme">dmz-cursor-theme package</a>, ran <code>update-alternatives --config x-cursor-theme</code>
and selected <code>/usr/share/icons/DMZ-Black/cursor.theme</code> as the cursor theme.</p>
<p>Overall, these displays are much brighter and more readable than the ones they
replaced.</p>Git packaging workflow for py-lmdb2015-07-05T00:56:00+00:00Robert Edmondstag:blog.mycre.ws,2015-07-05:articles/git-packaging-workflow-for-py-lmdb/<p>Recently, I packaged the <a href="https://lmdb.readthedocs.org/en/release/">py-lmdb Python binding</a> for the <a href="http://symas.com/mdb/">LMDB database
library</a>. This package is going to be team maintained by the <a href="https://alioth.debian.org/projects/pkg-db/">pkg-db group</a>,
which is responsible for maintaining BerkeleyDB and LMDB packages. Below are my
notes on (re-)Debianizing this package and how the Git repository for the source
package is laid out.</p>
<p>The upstream <code>py-lmdb</code> developer has a Git-centric workflow. Development is
done on the <code>master</code> branch, with regular releases done as fast-forward merges
to the <code>release</code> branch. Release tags of the form <code>py-lmdb_X.YZ</code> are provided.
The only tarballs provided are the ones that GitHub automatically generates
from tags. Since these tarballs are synthetic and the content of these
tarballs matches the content on the corresponding tag, we will ignore them in
favor of using the release tags directly. (The <code>--git-pristine-tar-commit</code>
option to <code>gbp-buildpackage</code> will be used so that <code>.orig.tar.gz</code> files can be
replicated so that the Debian archive will <a href="http://thread.gmane.org/gmane.linux.debian.devel.mentors/59711/focus=59721">accept subsequent uploads</a>,
but tarballs are otherwise irrelevant to our workflow.)</p>
<p>To make it clear that the release tags come from upstream's repository, they
should be prefixed with <code>upstream/</code>, which would preferably result in a <a href="http://dep.debian.net/deps/dep14/">DEP-14</a>
compliant scheme. (Unfortunately, since upstream's release tags begin with
<code>py-lmdb_</code>, this doesn't quite match the pattern that DEP-14 recommends.)</p>
<p>Here is how the local packaging repository is initialized. Note that <code>git clone</code>
isn't used, so that we can customize how the tags are fetched. Instead, we
create an empty Git repository and add the upstream repository as the <code>upstream</code>
remote. The <code>--no-tags</code> option is used, so that <code>git fetch</code> does not import the
remote's tags. However, we also add a custom fetch refspec
<code>refs/tags/*:refs/tags/upstream/*</code> so that the remote's tags are explicitly
fetched, but with the <code>upstream/</code> prefix.</p>
<pre><code class="boxed">$ mkdir py-lmdb
$ cd py-lmdb
$ git init
Initialized empty Git repository in /home/edmonds/debian/py-lmdb/.git/
$ git remote add --no-tags upstream https://github.com/dw/py-lmdb
$ git config --add remote.upstream.fetch 'refs/tags/*:refs/tags/upstream/*'
$ git fetch upstream
remote: Counting objects: 3336, done.
remote: Total 3336 (delta 0), reused 0 (delta 0), pack-reused 3336
Receiving objects: 100% (3336/3336), 2.15 MiB | 0 bytes/s, done.
Resolving deltas: 100% (1958/1958), done.
From https://github.com/dw/py-lmdb
* [new branch] master -> upstream/master
* [new branch] release -> upstream/release
* [new branch] win32-sparse-patch -> upstream/win32-sparse-patch
* [new tag] last-cython-version -> upstream/last-cython-version
* [new tag] py-lmdb_0.1 -> upstream/py-lmdb_0.1
* [new tag] py-lmdb_0.2 -> upstream/py-lmdb_0.2
* [new tag] py-lmdb_0.3 -> upstream/py-lmdb_0.3
* [new tag] py-lmdb_0.4 -> upstream/py-lmdb_0.4
* [new tag] py-lmdb_0.5 -> upstream/py-lmdb_0.5
* [new tag] py-lmdb_0.51 -> upstream/py-lmdb_0.51
* [new tag] py-lmdb_0.52 -> upstream/py-lmdb_0.52
* [new tag] py-lmdb_0.53 -> upstream/py-lmdb_0.53
* [new tag] py-lmdb_0.54 -> upstream/py-lmdb_0.54
* [new tag] py-lmdb_0.56 -> upstream/py-lmdb_0.56
* [new tag] py-lmdb_0.57 -> upstream/py-lmdb_0.57
* [new tag] py-lmdb_0.58 -> upstream/py-lmdb_0.58
* [new tag] py-lmdb_0.59 -> upstream/py-lmdb_0.59
* [new tag] py-lmdb_0.60 -> upstream/py-lmdb_0.60
* [new tag] py-lmdb_0.61 -> upstream/py-lmdb_0.61
* [new tag] py-lmdb_0.62 -> upstream/py-lmdb_0.62
* [new tag] py-lmdb_0.63 -> upstream/py-lmdb_0.63
* [new tag] py-lmdb_0.64 -> upstream/py-lmdb_0.64
* [new tag] py-lmdb_0.65 -> upstream/py-lmdb_0.65
* [new tag] py-lmdb_0.66 -> upstream/py-lmdb_0.66
* [new tag] py-lmdb_0.67 -> upstream/py-lmdb_0.67
* [new tag] py-lmdb_0.68 -> upstream/py-lmdb_0.68
* [new tag] py-lmdb_0.69 -> upstream/py-lmdb_0.69
* [new tag] py-lmdb_0.70 -> upstream/py-lmdb_0.70
* [new tag] py-lmdb_0.71 -> upstream/py-lmdb_0.71
* [new tag] py-lmdb_0.72 -> upstream/py-lmdb_0.72
* [new tag] py-lmdb_0.73 -> upstream/py-lmdb_0.73
* [new tag] py-lmdb_0.74 -> upstream/py-lmdb_0.74
* [new tag] py-lmdb_0.75 -> upstream/py-lmdb_0.75
* [new tag] py-lmdb_0.76 -> upstream/py-lmdb_0.76
* [new tag] py-lmdb_0.77 -> upstream/py-lmdb_0.77
* [new tag] py-lmdb_0.78 -> upstream/py-lmdb_0.78
* [new tag] py-lmdb_0.79 -> upstream/py-lmdb_0.79
* [new tag] py-lmdb_0.80 -> upstream/py-lmdb_0.80
* [new tag] py-lmdb_0.81 -> upstream/py-lmdb_0.81
* [new tag] py-lmdb_0.82 -> upstream/py-lmdb_0.82
* [new tag] py-lmdb_0.83 -> upstream/py-lmdb_0.83
* [new tag] py-lmdb_0.84 -> upstream/py-lmdb_0.84
* [new tag] py-lmdb_0.85 -> upstream/py-lmdb_0.85
* [new tag] py-lmdb_0.86 -> upstream/py-lmdb_0.86
$
</code></pre>
<p>Note that at this point we have content from the upstream remote in our local
repository, but we don't have any local branches:</p>
<pre><code class="boxed">$ git status
On branch master
Initial commit
nothing to commit (create/copy files and use "git add" to track)
$ git branch -a
remotes/upstream/master
remotes/upstream/release
remotes/upstream/win32-sparse-patch
$
</code></pre>
<p>We will use the <a href="http://dep.debian.net/deps/dep14/">DEP-14</a> naming scheme for the packaging branches, so the
branch for packages targeted at <code>unstable</code> will be called <code>debian/sid</code>. Since I
already made an <a href="https://tracker.debian.org/news/689247">initial 0.84-1 upload</a>, we need to start the <code>debian/sid</code>
branch from the upstream 0.84 tag and import the original packaging content
from that upload. The <code>--no-track</code> flag is passed to <code>git checkout</code> initially
so that Git doesn't consider the upstream release tag <code>upstream/py-lmdb_0.84</code>
to be the upstream branch for our packaging branch.</p>
<pre><code class="boxed">$ git checkout --no-track -b debian/sid upstream/py-lmdb_0.84
Switched to a new branch 'debian/sid'
$
</code></pre>
<p>At this point I imported the original packaging content for 0.84-1 with
<code>git am</code>. Then, I signed the <code>debian/0.84-1</code> tag:</p>
<pre><code class="boxed">$ git tag -s -m 'Debian release 0.84-1' debian/0.84-1
$ git verify-tag debian/0.84-1
gpg: Signature made Sat 04 Jul 2015 02:49:42 PM EDT using RSA key ID AAF6CDAE
gpg: Good signature from "Robert Edmonds <edmonds@mycre.ws>" [ultimate]
gpg: aka "Robert Edmonds <edmonds@fsi.io>" [ultimate]
gpg: aka "Robert Edmonds <edmonds@debian.org>" [ultimate]
$
</code></pre>
<p>New upstream releases are integrated by fetching new upstream tags and
non-fast-forward merging into the packaging branch. The latest release is 0.86,
so we merge from the <code>upstream/py-lmdb_0.86</code> tag.</p>
<pre><code class="boxed">$ git fetch upstream --dry-run
[...]
$ git fetch upstream
[...]
$ git checkout debian/sid
Already on 'debian/sid'
$ git merge --no-ff --no-edit upstream/py-lmdb_0.86
Merge made by the 'recursive' strategy.
ChangeLog | 46 ++++++++++++++
docs/index.rst | 46 +++++++++++++-
docs/themes/acid/layout.html | 4 +-
examples/dirtybench-gdbm.py | 6 ++
examples/dirtybench.py | 19 ++++++
examples/nastybench.py | 18 ++++--
examples/parabench.py | 6 ++
lib/lmdb.h | 37 ++++++-----
lib/mdb.c | 281 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------------------
lib/midl.c | 2 +-
lib/midl.h | 2 +-
lib/py-lmdb/preload.h | 48 ++++++++++++++
lmdb/__init__.py | 2 +-
lmdb/cffi.py | 120 ++++++++++++++++++++++++-----------
lmdb/cpython.c | 86 +++++++++++++++++++------
lmdb/tool.py | 5 +-
misc/gdb.commands | 21 ++++++
misc/runtests-travisci.sh | 3 +-
misc/runtests-ubuntu-12-04.sh | 28 ++++----
setup.py | 2 +
tests/crash_test.py | 22 +++++++
tests/cursor_test.py | 37 +++++++++++
tests/env_test.py | 73 +++++++++++++++++++++
tests/testlib.py | 14 +++-
tests/txn_test.py | 20 ++++++
25 files changed, 773 insertions(+), 175 deletions(-)
create mode 100644 lib/py-lmdb/preload.h
create mode 100644 misc/gdb.commands
$
</code></pre>
<p>Here I did some additional development work like editing the <code>debian/gbp.conf</code>
file and applying a fix for <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=790738">#790738</a> to make the package build reproducibly.
The package is now ready for an <a href="https://tracker.debian.org/news/695543">0.86-1 upload</a>, so I ran the following
<code>gbp dch</code> command:</p>
<pre><code class="boxed">$ gbp dch --release --auto --new-version=0.86-1 --commit
gbp:info: Found tag for topmost changelog version '6bdbb56c04571fe2d5d22aa0287ab0dc83959de5'
gbp:info: Continuing from commit '6bdbb56c04571fe2d5d22aa0287ab0dc83959de5'
gbp:info: Changelog has been committed for version 0.86-1
$
</code></pre>
<p>This automatically generates a changelog entry for 0.86-1, but it includes
commit summaries for all of the upstream commits since the last release, which I
had to edit out.</p>
<p>Then, I used <code>gbp buildpackage</code> with <code>BUILDER=pbuilder</code> to build the package in
a clean, up-to-date <code>sid</code> chroot. After checking the result, I signed the
<code>debian/0.86-1</code> tag:</p>
<pre><code class="boxed">$ git tag -s -m 'Debian release 0.86-1' debian/0.86-1
$
</code></pre>
<p>The package is now ready to be pushed to <code>git.debian.org</code>. First, a <a href="https://wiki.debian.org/Alioth/Git#Separate_project">bare
repository is initialized</a>:</p>
<pre><code class="boxed">$ ssh git.debian.org
edmonds@moszumanska:~$ cd /srv/git.debian.org/git/pkg-db/
edmonds@moszumanska:/srv/git.debian.org/git/pkg-db$ umask 002
edmonds@moszumanska:/srv/git.debian.org/git/pkg-db$ mkdir py-lmdb.git
edmonds@moszumanska:/srv/git.debian.org/git/pkg-db$ cd py-lmdb.git/
edmonds@moszumanska:/srv/git.debian.org/git/pkg-db/py-lmdb.git$ git --bare init --shared
Initialized empty shared Git repository in /srv/git.debian.org/git/pkg-db/py-lmdb.git/
edmonds@moszumanska:/srv/git.debian.org/git/pkg-db/py-lmdb.git$ echo 'py-lmdb Debian packaging' > description
edmonds@moszumanska:/srv/git.debian.org/git/pkg-db/py-lmdb.git$ mv hooks/post-update.sample hooks/post-update
edmonds@moszumanska:/srv/git.debian.org/git/pkg-db/py-lmdb.git$ chmod a+x hooks/post-update
edmonds@moszumanska:/srv/git.debian.org/git/pkg-db/py-lmdb.git$ logout
Shared connection to git.debian.org closed.
</code></pre>
<p>Then, we add a new <code>debian</code> remote to our local packaging repository. Per our
<a href="http://anonscm.debian.org/cgit/pkg-db/py-lmdb.git/plain/debian/README.source">repository conventions</a>, we need to ensure that only branch names matching
<code>debian/*</code> and <code>pristine-tar</code> and tag names matching <code>debian/*</code> and
<code>upstream/*</code> are pushed to the <code>debian</code> remote when we run <code>git push debian</code>,
so we add a a set of <code>remote.debian.push</code> refspecs that correspond to these
conventions. We also add an explicit <code>remote.debian.fetch</code> refspec to fetch
tags.</p>
<pre><code class="boxed">$ git remote add debian ssh://git.debian.org/git/pkg-db/py-lmdb.git
$ git config --add remote.debian.push 'refs/tags/debian/*'
$ git config --add remote.debian.push 'refs/tags/upstream/*'
$ git config --add remote.debian.push 'refs/heads/debian/*'
$ git config --add remote.debian.push 'refs/heads/pristine-tar'
$ git config --add remote.debian.fetch 'refs/tags/*:refs/tags/*'
</code></pre>
<p>We now run the initial push to the remote Git repository. The <code>--set-upstream</code>
option is used so that our local branches will be configured to track the
corresponding remote branches. Also note that the <code>debian/*</code> and <code>upstream/*</code>
tags are pushed as well.</p>
<pre><code class="boxed">$ git push debian --set-upstream
Counting objects: 3333, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (1083/1083), done.
Writing objects: 100% (3333/3333), 1.37 MiB | 0 bytes/s, done.
Total 3333 (delta 2231), reused 3314 (delta 2218)
To ssh://git.debian.org/git/pkg-db/py-lmdb.git
* [new branch] pristine-tar -> pristine-tar
* [new branch] debian/sid -> debian/sid
* [new tag] debian/0.84-1 -> debian/0.84-1
* [new tag] debian/0.86-1 -> debian/0.86-1
* [new tag] upstream/last-cython-version -> upstream/last-cython-version
* [new tag] upstream/py-lmdb_0.1 -> upstream/py-lmdb_0.1
* [new tag] upstream/py-lmdb_0.2 -> upstream/py-lmdb_0.2
* [new tag] upstream/py-lmdb_0.3 -> upstream/py-lmdb_0.3
* [new tag] upstream/py-lmdb_0.4 -> upstream/py-lmdb_0.4
* [new tag] upstream/py-lmdb_0.5 -> upstream/py-lmdb_0.5
* [new tag] upstream/py-lmdb_0.51 -> upstream/py-lmdb_0.51
* [new tag] upstream/py-lmdb_0.52 -> upstream/py-lmdb_0.52
* [new tag] upstream/py-lmdb_0.53 -> upstream/py-lmdb_0.53
* [new tag] upstream/py-lmdb_0.54 -> upstream/py-lmdb_0.54
* [new tag] upstream/py-lmdb_0.56 -> upstream/py-lmdb_0.56
* [new tag] upstream/py-lmdb_0.57 -> upstream/py-lmdb_0.57
* [new tag] upstream/py-lmdb_0.58 -> upstream/py-lmdb_0.58
* [new tag] upstream/py-lmdb_0.59 -> upstream/py-lmdb_0.59
* [new tag] upstream/py-lmdb_0.60 -> upstream/py-lmdb_0.60
* [new tag] upstream/py-lmdb_0.61 -> upstream/py-lmdb_0.61
* [new tag] upstream/py-lmdb_0.62 -> upstream/py-lmdb_0.62
* [new tag] upstream/py-lmdb_0.63 -> upstream/py-lmdb_0.63
* [new tag] upstream/py-lmdb_0.64 -> upstream/py-lmdb_0.64
* [new tag] upstream/py-lmdb_0.65 -> upstream/py-lmdb_0.65
* [new tag] upstream/py-lmdb_0.66 -> upstream/py-lmdb_0.66
* [new tag] upstream/py-lmdb_0.67 -> upstream/py-lmdb_0.67
* [new tag] upstream/py-lmdb_0.68 -> upstream/py-lmdb_0.68
* [new tag] upstream/py-lmdb_0.69 -> upstream/py-lmdb_0.69
* [new tag] upstream/py-lmdb_0.70 -> upstream/py-lmdb_0.70
* [new tag] upstream/py-lmdb_0.71 -> upstream/py-lmdb_0.71
* [new tag] upstream/py-lmdb_0.72 -> upstream/py-lmdb_0.72
* [new tag] upstream/py-lmdb_0.73 -> upstream/py-lmdb_0.73
* [new tag] upstream/py-lmdb_0.74 -> upstream/py-lmdb_0.74
* [new tag] upstream/py-lmdb_0.75 -> upstream/py-lmdb_0.75
* [new tag] upstream/py-lmdb_0.76 -> upstream/py-lmdb_0.76
* [new tag] upstream/py-lmdb_0.77 -> upstream/py-lmdb_0.77
* [new tag] upstream/py-lmdb_0.78 -> upstream/py-lmdb_0.78
* [new tag] upstream/py-lmdb_0.79 -> upstream/py-lmdb_0.79
* [new tag] upstream/py-lmdb_0.80 -> upstream/py-lmdb_0.80
* [new tag] upstream/py-lmdb_0.81 -> upstream/py-lmdb_0.81
* [new tag] upstream/py-lmdb_0.82 -> upstream/py-lmdb_0.82
* [new tag] upstream/py-lmdb_0.83 -> upstream/py-lmdb_0.83
* [new tag] upstream/py-lmdb_0.84 -> upstream/py-lmdb_0.84
* [new tag] upstream/py-lmdb_0.85 -> upstream/py-lmdb_0.85
* [new tag] upstream/py-lmdb_0.86 -> upstream/py-lmdb_0.86
Branch pristine-tar set up to track remote branch pristine-tar from debian.
Branch debian/sid set up to track remote branch debian/sid from debian.
$
</code></pre>
<p>After the initial push, we need to configure the remote repository so that
clones will checkout the <code>debian/sid</code> branch by default:</p>
<pre><code class="boxed">$ ssh git.debian.org
edmonds@moszumanska:~$ cd /srv/git.debian.org/git/pkg-db/py-lmdb.git/
edmonds@moszumanska:/srv/git.debian.org/git/pkg-db/py-lmdb.git$ git symbolic-ref HEAD refs/heads/debian/sid
edmonds@moszumanska:/srv/git.debian.org/git/pkg-db/py-lmdb.git$ logout
Shared connection to git.debian.org closed.
</code></pre>
<p>We can check if there are any updates in upstream's Git repository with the
following command:</p>
<pre><code class="boxed">$ git fetch upstream --dry-run -v
From https://github.com/dw/py-lmdb
= [up to date] master -> upstream/master
= [up to date] release -> upstream/release
= [up to date] win32-sparse-patch -> upstream/win32-sparse-patch
= [up to date] last-cython-version -> upstream/last-cython-version
= [up to date] py-lmdb_0.1 -> upstream/py-lmdb_0.1
= [up to date] py-lmdb_0.2 -> upstream/py-lmdb_0.2
= [up to date] py-lmdb_0.3 -> upstream/py-lmdb_0.3
= [up to date] py-lmdb_0.4 -> upstream/py-lmdb_0.4
= [up to date] py-lmdb_0.5 -> upstream/py-lmdb_0.5
= [up to date] py-lmdb_0.51 -> upstream/py-lmdb_0.51
= [up to date] py-lmdb_0.52 -> upstream/py-lmdb_0.52
= [up to date] py-lmdb_0.53 -> upstream/py-lmdb_0.53
= [up to date] py-lmdb_0.54 -> upstream/py-lmdb_0.54
= [up to date] py-lmdb_0.56 -> upstream/py-lmdb_0.56
= [up to date] py-lmdb_0.57 -> upstream/py-lmdb_0.57
= [up to date] py-lmdb_0.58 -> upstream/py-lmdb_0.58
= [up to date] py-lmdb_0.59 -> upstream/py-lmdb_0.59
= [up to date] py-lmdb_0.60 -> upstream/py-lmdb_0.60
= [up to date] py-lmdb_0.61 -> upstream/py-lmdb_0.61
= [up to date] py-lmdb_0.62 -> upstream/py-lmdb_0.62
= [up to date] py-lmdb_0.63 -> upstream/py-lmdb_0.63
= [up to date] py-lmdb_0.64 -> upstream/py-lmdb_0.64
= [up to date] py-lmdb_0.65 -> upstream/py-lmdb_0.65
= [up to date] py-lmdb_0.66 -> upstream/py-lmdb_0.66
= [up to date] py-lmdb_0.67 -> upstream/py-lmdb_0.67
= [up to date] py-lmdb_0.68 -> upstream/py-lmdb_0.68
= [up to date] py-lmdb_0.69 -> upstream/py-lmdb_0.69
= [up to date] py-lmdb_0.70 -> upstream/py-lmdb_0.70
= [up to date] py-lmdb_0.71 -> upstream/py-lmdb_0.71
= [up to date] py-lmdb_0.72 -> upstream/py-lmdb_0.72
= [up to date] py-lmdb_0.73 -> upstream/py-lmdb_0.73
= [up to date] py-lmdb_0.74 -> upstream/py-lmdb_0.74
= [up to date] py-lmdb_0.75 -> upstream/py-lmdb_0.75
= [up to date] py-lmdb_0.76 -> upstream/py-lmdb_0.76
= [up to date] py-lmdb_0.77 -> upstream/py-lmdb_0.77
= [up to date] py-lmdb_0.78 -> upstream/py-lmdb_0.78
= [up to date] py-lmdb_0.79 -> upstream/py-lmdb_0.79
= [up to date] py-lmdb_0.80 -> upstream/py-lmdb_0.80
= [up to date] py-lmdb_0.81 -> upstream/py-lmdb_0.81
= [up to date] py-lmdb_0.82 -> upstream/py-lmdb_0.82
= [up to date] py-lmdb_0.83 -> upstream/py-lmdb_0.83
= [up to date] py-lmdb_0.84 -> upstream/py-lmdb_0.84
= [up to date] py-lmdb_0.85 -> upstream/py-lmdb_0.85
= [up to date] py-lmdb_0.86 -> upstream/py-lmdb_0.86
</code></pre>
<p>We can check if any co-maintainers have pushed updates to the <code>git.debian.org</code>
repository with the following command:</p>
<pre><code class="boxed">$ git fetch debian --dry-run -v
From ssh://git.debian.org/git/pkg-db/py-lmdb
= [up to date] debian/sid -> debian/debian/sid
= [up to date] pristine-tar -> debian/pristine-tar
= [up to date] debian/0.84-1 -> debian/0.84-1
= [up to date] debian/0.86-1 -> debian/0.86-1
= [up to date] upstream/last-cython-version -> upstream/last-cython-version
= [up to date] upstream/py-lmdb_0.1 -> upstream/py-lmdb_0.1
= [up to date] upstream/py-lmdb_0.2 -> upstream/py-lmdb_0.2
= [up to date] upstream/py-lmdb_0.3 -> upstream/py-lmdb_0.3
= [up to date] upstream/py-lmdb_0.4 -> upstream/py-lmdb_0.4
= [up to date] upstream/py-lmdb_0.5 -> upstream/py-lmdb_0.5
= [up to date] upstream/py-lmdb_0.51 -> upstream/py-lmdb_0.51
= [up to date] upstream/py-lmdb_0.52 -> upstream/py-lmdb_0.52
= [up to date] upstream/py-lmdb_0.53 -> upstream/py-lmdb_0.53
= [up to date] upstream/py-lmdb_0.54 -> upstream/py-lmdb_0.54
= [up to date] upstream/py-lmdb_0.56 -> upstream/py-lmdb_0.56
= [up to date] upstream/py-lmdb_0.57 -> upstream/py-lmdb_0.57
= [up to date] upstream/py-lmdb_0.58 -> upstream/py-lmdb_0.58
= [up to date] upstream/py-lmdb_0.59 -> upstream/py-lmdb_0.59
= [up to date] upstream/py-lmdb_0.60 -> upstream/py-lmdb_0.60
= [up to date] upstream/py-lmdb_0.61 -> upstream/py-lmdb_0.61
= [up to date] upstream/py-lmdb_0.62 -> upstream/py-lmdb_0.62
= [up to date] upstream/py-lmdb_0.63 -> upstream/py-lmdb_0.63
= [up to date] upstream/py-lmdb_0.64 -> upstream/py-lmdb_0.64
= [up to date] upstream/py-lmdb_0.65 -> upstream/py-lmdb_0.65
= [up to date] upstream/py-lmdb_0.66 -> upstream/py-lmdb_0.66
= [up to date] upstream/py-lmdb_0.67 -> upstream/py-lmdb_0.67
= [up to date] upstream/py-lmdb_0.68 -> upstream/py-lmdb_0.68
= [up to date] upstream/py-lmdb_0.69 -> upstream/py-lmdb_0.69
= [up to date] upstream/py-lmdb_0.70 -> upstream/py-lmdb_0.70
= [up to date] upstream/py-lmdb_0.71 -> upstream/py-lmdb_0.71
= [up to date] upstream/py-lmdb_0.72 -> upstream/py-lmdb_0.72
= [up to date] upstream/py-lmdb_0.73 -> upstream/py-lmdb_0.73
= [up to date] upstream/py-lmdb_0.74 -> upstream/py-lmdb_0.74
= [up to date] upstream/py-lmdb_0.75 -> upstream/py-lmdb_0.75
= [up to date] upstream/py-lmdb_0.76 -> upstream/py-lmdb_0.76
= [up to date] upstream/py-lmdb_0.77 -> upstream/py-lmdb_0.77
= [up to date] upstream/py-lmdb_0.78 -> upstream/py-lmdb_0.78
= [up to date] upstream/py-lmdb_0.79 -> upstream/py-lmdb_0.79
= [up to date] upstream/py-lmdb_0.80 -> upstream/py-lmdb_0.80
= [up to date] upstream/py-lmdb_0.81 -> upstream/py-lmdb_0.81
= [up to date] upstream/py-lmdb_0.82 -> upstream/py-lmdb_0.82
= [up to date] upstream/py-lmdb_0.83 -> upstream/py-lmdb_0.83
= [up to date] upstream/py-lmdb_0.84 -> upstream/py-lmdb_0.84
= [up to date] upstream/py-lmdb_0.85 -> upstream/py-lmdb_0.85
= [up to date] upstream/py-lmdb_0.86 -> upstream/py-lmdb_0.86
$
</code></pre>
<p>We can check if anything needs to be pushed from our local repository to the
<code>git.debian.org</code> repository with the following command:</p>
<pre><code class="boxed">$ git push debian --dry-run -v
Pushing to ssh://git.debian.org/git/pkg-db/py-lmdb.git
To ssh://git.debian.org/git/pkg-db/py-lmdb.git
= [up to date] debian/sid -> debian/sid
= [up to date] pristine-tar -> pristine-tar
= [up to date] debian/0.84-1 -> debian/0.84-1
= [up to date] debian/0.86-1 -> debian/0.86-1
= [up to date] upstream/last-cython-version -> upstream/last-cython-version
= [up to date] upstream/py-lmdb_0.1 -> upstream/py-lmdb_0.1
= [up to date] upstream/py-lmdb_0.2 -> upstream/py-lmdb_0.2
= [up to date] upstream/py-lmdb_0.3 -> upstream/py-lmdb_0.3
= [up to date] upstream/py-lmdb_0.4 -> upstream/py-lmdb_0.4
= [up to date] upstream/py-lmdb_0.5 -> upstream/py-lmdb_0.5
= [up to date] upstream/py-lmdb_0.51 -> upstream/py-lmdb_0.51
= [up to date] upstream/py-lmdb_0.52 -> upstream/py-lmdb_0.52
= [up to date] upstream/py-lmdb_0.53 -> upstream/py-lmdb_0.53
= [up to date] upstream/py-lmdb_0.54 -> upstream/py-lmdb_0.54
= [up to date] upstream/py-lmdb_0.56 -> upstream/py-lmdb_0.56
= [up to date] upstream/py-lmdb_0.57 -> upstream/py-lmdb_0.57
= [up to date] upstream/py-lmdb_0.58 -> upstream/py-lmdb_0.58
= [up to date] upstream/py-lmdb_0.59 -> upstream/py-lmdb_0.59
= [up to date] upstream/py-lmdb_0.60 -> upstream/py-lmdb_0.60
= [up to date] upstream/py-lmdb_0.61 -> upstream/py-lmdb_0.61
= [up to date] upstream/py-lmdb_0.62 -> upstream/py-lmdb_0.62
= [up to date] upstream/py-lmdb_0.63 -> upstream/py-lmdb_0.63
= [up to date] upstream/py-lmdb_0.64 -> upstream/py-lmdb_0.64
= [up to date] upstream/py-lmdb_0.65 -> upstream/py-lmdb_0.65
= [up to date] upstream/py-lmdb_0.66 -> upstream/py-lmdb_0.66
= [up to date] upstream/py-lmdb_0.67 -> upstream/py-lmdb_0.67
= [up to date] upstream/py-lmdb_0.68 -> upstream/py-lmdb_0.68
= [up to date] upstream/py-lmdb_0.69 -> upstream/py-lmdb_0.69
= [up to date] upstream/py-lmdb_0.70 -> upstream/py-lmdb_0.70
= [up to date] upstream/py-lmdb_0.71 -> upstream/py-lmdb_0.71
= [up to date] upstream/py-lmdb_0.72 -> upstream/py-lmdb_0.72
= [up to date] upstream/py-lmdb_0.73 -> upstream/py-lmdb_0.73
= [up to date] upstream/py-lmdb_0.74 -> upstream/py-lmdb_0.74
= [up to date] upstream/py-lmdb_0.75 -> upstream/py-lmdb_0.75
= [up to date] upstream/py-lmdb_0.76 -> upstream/py-lmdb_0.76
= [up to date] upstream/py-lmdb_0.77 -> upstream/py-lmdb_0.77
= [up to date] upstream/py-lmdb_0.78 -> upstream/py-lmdb_0.78
= [up to date] upstream/py-lmdb_0.79 -> upstream/py-lmdb_0.79
= [up to date] upstream/py-lmdb_0.80 -> upstream/py-lmdb_0.80
= [up to date] upstream/py-lmdb_0.81 -> upstream/py-lmdb_0.81
= [up to date] upstream/py-lmdb_0.82 -> upstream/py-lmdb_0.82
= [up to date] upstream/py-lmdb_0.83 -> upstream/py-lmdb_0.83
= [up to date] upstream/py-lmdb_0.84 -> upstream/py-lmdb_0.84
= [up to date] upstream/py-lmdb_0.85 -> upstream/py-lmdb_0.85
= [up to date] upstream/py-lmdb_0.86 -> upstream/py-lmdb_0.86
Everything up-to-date
</code></pre>
<p>Finally, in order to set up a fresh local clone of the <code>git.debian.org</code>
repository that's configured like the local repository created above, we have
to do the following:</p>
<pre><code class="boxed">$ git clone --origin debian ssh://git.debian.org/git/pkg-db/py-lmdb.git
Cloning into 'py-lmdb'...
remote: Counting objects: 3333, done.
remote: Compressing objects: 100% (1070/1070), done.
remote: Total 3333 (delta 2231), reused 3333 (delta 2231)
Receiving objects: 100% (3333/3333), 1.37 MiB | 1.11 MiB/s, done.
Resolving deltas: 100% (2231/2231), done.
Checking connectivity... done.
$ cd py-lmdb
$ git remote add --no-tags upstream https://github.com/dw/py-lmdb
$ git config --add remote.upstream.fetch 'refs/tags/*:refs/tags/upstream/*'
$ git fetch upstream
remote: Counting objects: 56, done.
remote: Total 56 (delta 25), reused 25 (delta 25), pack-reused 31
Unpacking objects: 100% (56/56), done.
From https://github.com/dw/py-lmdb
* [new branch] master -> upstream/master
* [new branch] release -> upstream/release
* [new branch] win32-sparse-patch -> upstream/win32-sparse-patch
$ git branch --track pristine-tar debian/pristine-tar
Branch pristine-tar set up to track remote branch pristine-tar from debian.
$ git config --add remote.debian.push 'refs/tags/debian/*'
$ git config --add remote.debian.push 'refs/tags/upstream/*'
$ git config --add remote.debian.push 'refs/heads/debian/*'
$ git config --add remote.debian.push 'refs/heads/pristine-tar'
$ git config --add remote.debian.fetch 'refs/tags/*:refs/tags/*'
$
</code></pre>
<p>This is a fair amount of effort beyond a simple <code>git clone</code>, though, so I
wonder if anything can be done to optimize this.</p>Bad Google repository signatures2015-03-22T03:50:57+00:00Robert Edmondstag:blog.mycre.ws,2015-03-22:articles/bad-google-repository-signatures/<p><strong>Update</strong>: <em>I was able to get in touch with the Googlers responsible for the
<a href="http://talks.golang.org/2013/oscon-dl.slide">dl.google.com service</a>, and the root cause for the mismatched signature problem
described below has been found and fixed. I now consistently receive from
Google's servers <code>Release</code> and <code>Release.gpg</code> files that pass apt's signature
validation on my home connection.</em></p>
<p>Google publishes <a href="https://www.google.com/linuxrepositories/">Linux software repositories</a> for several of their products,
including Google Chrome, which is available from the following apt source:</p>
<pre><code class="boxed">deb http://dl.google.com/linux/chrome/deb/ stable main
</code></pre>
<p>These repositories are signed with an 8 year old 1024-bit DSA key:</p>
<pre><code class="boxed">pub 1024D/7FAC5991 2007-03-08
Key fingerprint = 4CCA 1EAF 950C EE4A B839 76DC A040 830F 7FAC 5991
uid Google, Inc. Linux Package Signing Key <linux-packages-keymaster@google.com>
sub 2048g/C07CB649 2007-03-08
</code></pre>
<p>Asymmetric 1024-bit keys are not considered strong enough and were, for
instance, <a href="http://googleonlinesecurity.blogspot.com/2013/11/out-with-old-stronger-certificates-with.html">aggressively retired</a> from Google's SSL frontends almost two years
ago. Such short keys should not be used to protect the integrity of software
package repositories.</p>
<p>Note that this key has a longer 2048-bit ElGamal subkey, which is not actually
used to produce signatures, but only for encryption. In fact, only a signing key
is needed to sign the files in a secure apt repository, and, for instance, the
<a href="https://ftp-master.debian.org/keys.html">archive keys</a> used to sign official debian.org repositories do not contain an
encryption subkey.</p>
<p>Since years, many users have reported an <a href="https://www.google.com/search?q=%22BADSIG+A040830F7FAC5991%22">error message</a> like the following when
running <code>apt-get update</code>:</p>
<pre><code class="boxed">W: GPG error: http://dl.google.com stable Release: The following signatures were
invalid: BADSIG A040830F7FAC5991 Google, Inc. Linux Package Signing Key
<linux-packages-keymaster@google.com>
</code></pre>
<p>This error might resolve itself if <code>apt-get update</code> is run again. Apparently,
this is due to "<a href="https://code.google.com/p/chromium/issues/detail?id=107334#c12">bad pushes</a>" occurring in the Google infrastructure. An example
of this can be seen in the following <code>curl</code> output:</p>
<pre><code class="boxed">$ curl -v http://dl.google.com/linux/chrome/deb/dists/stable/Release \
http://dl.google.com/linux/chrome/deb/dists/stable/Release.gpg
* Hostname was NOT found in DNS cache
* Trying 74.125.196.136...
* Connected to dl.google.com (74.125.196.136) port 80 (#0)
> GET /linux/chrome/deb/dists/stable/Release HTTP/1.1
> User-Agent: curl/7.38.0
> Host: dl.google.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 1347
< Content-Type: application/octet-stream
< Etag: "518b8"
< Expires: Sun, 22 Mar 2015 18:55:19 PDT
< Last-Modified: Fri, 20 Mar 2015 04:22:00 GMT
* Server downloads is not blacklisted
< Server: downloads
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Xss-Protection: 1; mode=block
< Date: Sun, 22 Mar 2015 01:55:19 GMT
< Alternate-Protocol: 80:quic,p=0.5
<
Origin: Google, Inc.
Label: Google
Suite: stable
Codename: stable
Version: 1.0
Date: Thu, 19 Mar 2015 22:55:29 +0000
Architectures: amd64 i386
Components: main
Description: Google chrome-linux repository.
MD5Sum:
53375c7a2d182d85aef6218c179040ed 144 main/binary-i386/Release
c556daf52ac818e4b11b84cb5943f6e0 4076 main/binary-i386/Packages
867ba456bd6537e51bd344df212f4662 960 main/binary-i386/Packages.gz
2b766b2639b57d5282a154cf6a00b172 1176 main/binary-i386/Packages.bz2
89704f9af9e6ccd87c192de11ba4c511 145 main/binary-amd64/Release
fa88101278271922ec9b14b030fd2423 4082 main/binary-amd64/Packages
1ba717117027f36ff4aea9c3ea60de9e 962 main/binary-amd64/Packages.gz
19af18f376c986d317cadb3394c60ac5 1193 main/binary-amd64/Packages.bz2
SHA1:
59414c4175f2cc22e67ba6c30687b00c72a7eafc 144 main/binary-i386/Release
1764c5418478b1077ada54c73eb501165ba79170 4076 main/binary-i386/Packages
db24eafac51d3e63fd41343028fb3243f96cbed6 960 main/binary-i386/Packages.gz
ad8be07425e88b2fdf2f6d143989cde1341a8c51 1176 main/binary-i386/Packages.bz2
153199d8f866350b7853365a4adc95ee687603dd 145 main/binary-amd64/Release
7ce66535b35d5fc267fe23af9947f9d27e88508b 4082 main/binary-amd64/Packages
a72b5e46c3be8ad403df54e4cdcd6e58b2ede65a 962 main/binary-amd64/Packages.gz
dbc7fddd28cc742ef8f0fb8c6e096455e18c35f8 1193 main/binary-amd64/Packages.bz2
* Connection #0 to host dl.google.com left intact
* Found bundle for host dl.google.com: 0x7f24e68d06a0
* Re-using existing connection! (#0) with host dl.google.com
* Connected to dl.google.com (74.125.196.136) port 80 (#0)
> GET /linux/chrome/deb/dists/stable/Release.gpg HTTP/1.1
> User-Agent: curl/7.38.0
> Host: dl.google.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 198
< Content-Type: application/octet-stream
< Etag: "518f4"
< Expires: Sun, 22 Mar 2015 18:55:19 PDT
< Last-Modified: Fri, 20 Mar 2015 04:05:00 GMT
* Server downloads is not blacklisted
< Server: downloads
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Xss-Protection: 1; mode=block
< Date: Sun, 22 Mar 2015 01:55:19 GMT
< Alternate-Protocol: 80:quic,p=0.5
<
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
iEYEABECAAYFAlULm7YACgkQoECDD3+sWZFyxACeNPuK/zQ0v+3Py1n2s09Wk/Ti
DckAni8V/gy++xIinu8OdUXv7c777V9H
=5vT6
-----END PGP SIGNATURE-----
* Connection #0 to host dl.google.com left intact
</code></pre>
<p>Note that both the <code>Release</code> and <code>Release.gpg</code> files were fetched with the same
HTTP connection, so the two files must have come from the same web frontend.
(Though, it is possible they were served by different backends.) However, the
detached signature in <code>Release.gpg</code> does not match the content in <code>Release</code>:</p>
<pre><code class="boxed">gpgv: Signature made Fri 20 Mar 2015 12:01:58 AM EDT using DSA key ID 7FAC5991
gpgv: BAD signature from "Google, Inc. Linux Package Signing Key <linux-packages-keymaster@google.com>"
</code></pre>
<p>Performing the same pair of fetches again, the same <code>Release.gpg</code> file is
returned, but the <code>Release</code> file is slightly different:</p>
<pre><code class="boxed">$ curl -v http://dl.google.com/linux/chrome/deb/dists/stable/Release \
http://dl.google.com/linux/chrome/deb/dists/stable/Release.gpg
* Hostname was NOT found in DNS cache
* Trying 74.125.196.136...
* Connected to dl.google.com (74.125.196.136) port 80 (#0)
> GET /linux/chrome/deb/dists/stable/Release HTTP/1.1
> User-Agent: curl/7.38.0
> Host: dl.google.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 1347
< Content-Type: application/octet-stream
< Etag: "518f3"
< Expires: Sun, 22 Mar 2015 18:55:04 PDT
< Last-Modified: Fri, 20 Mar 2015 04:05:00 GMT
* Server downloads is not blacklisted
< Server: downloads
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Xss-Protection: 1; mode=block
< Date: Sun, 22 Mar 2015 01:55:04 GMT
< Alternate-Protocol: 80:quic,p=0.5
<
Origin: Google, Inc.
Label: Google
Suite: stable
Codename: stable
Version: 1.0
Date: Fri, 20 Mar 2015 04:02:02 +0000
Architectures: amd64 i386
Components: main
Description: Google chrome-linux repository.
MD5Sum:
89704f9af9e6ccd87c192de11ba4c511 145 main/binary-amd64/Release
fa88101278271922ec9b14b030fd2423 4082 main/binary-amd64/Packages
1ba717117027f36ff4aea9c3ea60de9e 962 main/binary-amd64/Packages.gz
19af18f376c986d317cadb3394c60ac5 1193 main/binary-amd64/Packages.bz2
53375c7a2d182d85aef6218c179040ed 144 main/binary-i386/Release
c556daf52ac818e4b11b84cb5943f6e0 4076 main/binary-i386/Packages
867ba456bd6537e51bd344df212f4662 960 main/binary-i386/Packages.gz
2b766b2639b57d5282a154cf6a00b172 1176 main/binary-i386/Packages.bz2
SHA1:
153199d8f866350b7853365a4adc95ee687603dd 145 main/binary-amd64/Release
7ce66535b35d5fc267fe23af9947f9d27e88508b 4082 main/binary-amd64/Packages
a72b5e46c3be8ad403df54e4cdcd6e58b2ede65a 962 main/binary-amd64/Packages.gz
dbc7fddd28cc742ef8f0fb8c6e096455e18c35f8 1193 main/binary-amd64/Packages.bz2
59414c4175f2cc22e67ba6c30687b00c72a7eafc 144 main/binary-i386/Release
1764c5418478b1077ada54c73eb501165ba79170 4076 main/binary-i386/Packages
db24eafac51d3e63fd41343028fb3243f96cbed6 960 main/binary-i386/Packages.gz
ad8be07425e88b2fdf2f6d143989cde1341a8c51 1176 main/binary-i386/Packages.bz2
* Connection #0 to host dl.google.com left intact
* Found bundle for host dl.google.com: 0x7ffa33d8b6a0
* Re-using existing connection! (#0) with host dl.google.com
* Connected to dl.google.com (74.125.196.136) port 80 (#0)
> GET /linux/chrome/deb/dists/stable/Release.gpg HTTP/1.1
> User-Agent: curl/7.38.0
> Host: dl.google.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 198
< Content-Type: application/octet-stream
< Etag: "518f4"
< Expires: Sun, 22 Mar 2015 18:55:05 PDT
< Last-Modified: Fri, 20 Mar 2015 04:05:00 GMT
* Server downloads is not blacklisted
< Server: downloads
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Xss-Protection: 1; mode=block
< Date: Sun, 22 Mar 2015 01:55:05 GMT
< Alternate-Protocol: 80:quic,p=0.5
<
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
iEYEABECAAYFAlULm7YACgkQoECDD3+sWZFyxACeNPuK/zQ0v+3Py1n2s09Wk/Ti
DckAni8V/gy++xIinu8OdUXv7c777V9H
=5vT6
-----END PGP SIGNATURE-----
* Connection #0 to host dl.google.com left intact
</code></pre>
<p>Note that the <code>Date</code> line in the <code>Release</code> file is different:</p>
<pre><code class="boxed">@@ -6 +6 @@
-Date: Thu, 19 Mar 2015 22:55:29 +0000
+Date: Fri, 20 Mar 2015 04:02:02 +0000
</code></pre>
<p>The file hashes listed in the <code>Release</code> file are in a different order, as well,
though the actual hash values are the same. This <code>Release</code> file <strong>does</strong> have a
valid signature:</p>
<pre><code class="boxed">gpgv: Signature made Fri 20 Mar 2015 12:01:58 AM EDT using DSA key ID 7FAC5991
gpgv: Good signature from "Google, Inc. Linux Package Signing Key <linux-packages-keymaster@google.com>"
</code></pre>
<p>Note that the <code>Release.gpg</code> files in the good and bad cases are the same, and
the same signature cannot cover two files with different content. Also note that
the same mis-signed content is <a href="https://dl.google.com/linux/chrome/deb/dists/stable/Release">available via HTTPS</a>, so it is probably not
caused by a MITM attack.</p>
<p>The possibility of skew between the <code>Release</code> and <code>Release.gpg</code> files is
precisely why <a href="https://lists.debian.org/debian-devel-announce/2009/11/msg00001.html">inline signed Release files</a> were introduced, but Google's
repositories use only the older format with a detached signature.</p>
<p>It would be nice if Google could fix the underlying bug in their infrastructure
that results in mis-signed repositories being published frequently, because it
trains users to ignore cryptographic failures.</p>Converting to --upstream-vcs-tag2015-03-01T22:36:32+00:00Robert Edmondstag:blog.mycre.ws,2015-03-01:articles/converting-to-upstream-vcs-tag/<p>Recently, the Google protobuf developers <a href="https://groups.google.com/d/msg/protobuf/3qvNnYo8-SM/GSMYGGBNXfoJ">announced</a> a migration of their
project's source code from an <a href="https://code.google.com/p/protobuf/source/browse/">svn repository</a> to a <a href="https://github.com/google/protobuf">git repository</a>. Up until
this point, the <a href="http://anonscm.debian.org/cgit/collab-maint/protobuf.git">Debian protobuf package repository</a> had only tracked upstream
development by embedding upstream release tarballs using <code>gbp import-orig</code> with
<code>pristine-tar</code>. It would be nice to smoothly migrate the packaging repository to
additionally make use of the <code>--upstream-vcs-tag</code> option to <code>gbp import-orig</code>,
the advantages of which have been <a href="http://www.eyrie.org/~eagle/journal/2013-04/001.html">well described by Russ Allbery</a>.</p>
<p>This turned out to be harder than expected, so for reference I documented the
steps I took below. Note that this packaging repository uses the default <code>gbp
import-orig</code> repository layout, where upstream sources are placed on a branch
named <code>upstream</code>, and the Debian branch is named <code>master</code>.</p>
<p>Add an <code>upstream</code> remote configured to track the upstream repository's <code>master</code>
branch and tags.</p>
<pre><code class="boxed">$ git remote add --tags --track master upstream https://github.com/google/protobuf.git
</code></pre>
<p>The <code>upstream</code> remote shouldn't be confused with our <code>upstream</code> branch. Note
that git-remotes are local to the repository, so the <code>upstream</code> remote should
probably be documented in the <code>debian/README.source</code> file.</p>
<p>Fetch the upstream branch and tags.</p>
<pre><code class="boxed">$ git fetch upstream
warning: no common commits
remote: Counting objects: 5210, done.
remote: Compressing objects: 100% (861/861), done.
remote: Total 5210 (delta 3869), reused 5194 (delta 3855)
Receiving objects: 100% (5210/5210), 3.57 MiB | 1.43 MiB/s, done.
Resolving deltas: 100% (3869/3869), done.
From https://github.com/google/protobuf
* [new branch] master -> upstream/master
* [new tag] v2.6.0 -> v2.6.0
$
</code></pre>
<p>We now have a git-remote <code>upstream</code>, a remote-tracking branch <code>upstream/master</code>
which corresponds to the <code>master</code> branch that upstream makes releases from, and
a release tag <code>v2.6.0</code>. Note that the remote-tracking branch <code>upstream/master</code>
shouldn't be confused with our <code>master</code> branch.</p>
<p>Up until this point, our <code>upstream</code> branch has been synthetically generated by
importing upstream's release tarballs with <code>gbp import-orig</code>. We need to merge
this synthetic history with <code>upstream/master</code>. Unfortunately, I couldn't find a
way to do this without using a temporary branch.</p>
<pre><code class="boxed">$ git checkout -b tmp upstream/master
Branch tmp set up to track remote branch master from upstream.
Switched to a new branch 'tmp'
$ git merge -s ours -m \
"Merge the original 'upstream' branch with upstream's new master branch" upstream
Merge made by the 'ours' strategy.
$ git checkout upstream
Switched to branch 'upstream'
Your branch is up-to-date with 'origin/upstream'.
$ git merge --ff-only tmp
Updating 7ed940b..9ba221e
Fast-forward
CHANGES.txt | 49 +-
COPYING.txt => LICENSE | 0
Makefile.am | 64 +-
Makefile.in | 1041 --
README.txt => README.md | 49 +-
[...many more lines...]
$ git branch -D tmp
Deleted branch tmp (was 5f18f02).
$
</code></pre>
<p>There are now an additional 400 or so commits on our <code>upstream</code> branch,
corresponding to the new git repository history published by upstream.</p>
<p>Import the 2.6.0 release tarball against the upstream <code>v2.6.0</code> tag, using the
<code>--upstream-vcs-tag</code> option.</p>
<pre><code class="boxed">$ git checkout master
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
$ gbp import-orig -u 2.6.0 --upstream-vcs-tag=v2.6.0 ~/debian/tarballs/protobuf_2.6.0.orig.tar.gz
gbp:info: Importing '/home/edmonds/debian/tarballs/protobuf_2.6.0.orig.tar.gz' to branch 'upstream'...
gbp:info: Source package is protobuf
gbp:info: Upstream version is 2.6.0
pristine-tar: committed protobuf_2.6.0.orig.tar.gz.delta to branch pristine-tar
gbp:info: Merging to 'master'
gbp:info: Successfully imported version 2.6.0 of /home/edmonds/debian/tarballs/protobuf_2.6.0.orig.tar.gz
$
</code></pre>
<p>The <code>upstream</code> branch now contains a mixture of the original series of release
tarball content imported by plain <code>gbp import-orig</code> and the <code>upstream/master</code>
branch as published by upstream.</p>
<p>Updating the Debian packaging repository when new upstream releases occur only
requires a <code>git fetch</code> to pull down upstream's updated git history and release
tag and using the <code>--upstream-vcs-tag</code> option when importing the release
tarball with <code>gbp import-orig</code>.</p>BIND and GCC 4.92014-06-06T02:19:08+00:00Robert Edmondstag:blog.mycre.ws,2014-06-06:articles/bind-and-gcc-49/<p>ISC has issued an <a href="https://kb.isc.org/article/AA-01167">operational notification</a> advising the use of the
<code>-fno-delete-null-pointer-checks</code> flag when compiling current versions of BIND
with GCC 4.9:</p>
<blockquote>
<p>Beginning with GCC 4.9.0, code optimization in GCC now includes (by default)
an optimization which is intended to eliminate unnecessary null pointer
comparisons in compiled code. Unfortunately this optimization removes checks
which are necessary in BIND and the demonstrated effect is to cause
unpredictable assertion failures during execution of named, resulting in
termination of the server process.</p>
<p>Future versions of BIND will be modified so that the optimizer does not
incorrectly remove necessary checks when building from source, and until those
versions are available multiple immediate workarounds are available.</p>
</blockquote>
<p>According to the <a href="https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html">GCC documentation</a>, <code>-fdelete-null-pointer-checks</code> performs
the following optimization, and is enabled by default even when compiling with
<code>-O0</code>:</p>
<blockquote>
<p>Assume that programs cannot safely dereference null pointers, and that no code
or data element resides there. This enables simple constant folding
optimizations at all optimization levels. In addition, other optimization
passes in GCC use this flag to control global dataflow analyses that eliminate
useless checks for null pointers; these assume that if a pointer is checked
after it has already been dereferenced, it cannot be null.</p>
</blockquote>
<p>This optimization <a href="http://www.gnu.org/software/gcc/news/null.html">dates back to 1999</a> but it recently <a href="https://gcc.gnu.org/gcc-4.9/porting_to.html">became more aggressive</a>
in the GCC 4.9 release:</p>
<blockquote>
<p>GCC might now optimize away the null pointer check in code like:</p>
<pre><code>int copy (int* dest, int* src, size_t nbytes) {
memmove (dest, src, nbytes);
if (src != NULL)
return *src;
return 0;
}
</code></pre>
<p>The pointers passed to <code>memmove</code> (and similar functions in <code><string.h></code>) must
be non-null even when <code>nbytes==0</code>, so GCC can use that information to remove
the check after the <code>memmove</code> call. Calling <code>copy(p, NULL, 0)</code> can therefore
deference a null pointer and crash.</p>
<p>The example above needs to be fixed to avoid the invalid <code>memmove</code> call, for
example:</p>
<pre><code>if (nbytes != 0)
memmove (dest, src, nbytes);
</code></pre>
</blockquote>
<p>It's interesting that the ISC operational advisory labels this an "incorrect"
optimization. I wonder if this is a similar case to the <a href="http://lwn.net/Articles/414467/">Glibc optimization</a>
which exposed incorrect usage of <code>memcpy()</code> in many applications, <a href="https://kb.isc.org/article/AA-01085">including
BIND</a>.</p>
<p>Fortunately, few users should be affected by this bug, since GCC 4.9 is fairly
new. (The upcoming <a href="http://fedoraproject.org/wiki/Releases/21/ChangeSet#GCC49">Fedora 21 release</a> will probably be the first mainstream
distribution to ship with GCC 4.9 as the default compiler.)</p>
<p><strong>Update</strong>: Debian bug <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=750760">#750760</a> filed.</p>