Filelists sqlite bz2 errno 14 http error 404 not found

Comment 1

Zdeněk Pavlas

2011-10-10 14:51:23 UTC

> fedora-chromium               | 3.4 kB     00:00     
.. so we grab a fresh repomd.xml

> fedora-chromium/primary_db    | 9.9 kB     00:00
.. primary metadata were there

> fedora-chromium/filelists_db   => 404
.. but not the filelist!

Hmm, this suggests there's something wrong with mirrors themselves.  I've already seen that few mirrors 404 on primary_db, right after frequent repomd.xml updates, but never all of them.

> IOW, the signature prefix on that file shouldn't be offered until the mirrors are ready to serve it, right?

When repo gets updated, repomd and MD files are very likely pushed independently, otherwise a single dead mirror would block the whole process.  Filelists tend to be large so it may take a while.  I'd suggest just try again in few minutes.

Just to make sure there's no bug in yum, please run
# egrep '(primary|filelists).sqlite' /var/cache/yum/fedora-chromium/repomd.xml 
# ls -l /var/cache/yum/fedora-chromium/*primary.sqlite

and check that both the retrieved *primary.sqlite and the filelist that gave 404 have used the same repomd.xml.

Comment 2

Jeff Layton

2011-10-10 15:06:39 UTC

I don't think the problem is with the chromium repo. The log says this at the end:

Error: failure: repodata/6ad3ddc0b0eb5b314f41c263f9625a9528ed699d8159bebdec56203ab736ccfe-filelists.sqlite.bz2 from fedora-debuginfo: [Errno 256] No more mirrors to try.

I haven't tried to update again since I generated the log, but here's the output you requested:

[root@tlielax ~]# egrep '(primary|filelists).sqlite' /var/cache/yum/fedora-chromium/repomd.xml 
  <location href="repodata/2edeb55260fe1e0fc9a00d6baf92bef74976f9393a82e7eb502c342a4bde2d38-primary.sqlite.bz2"/>
  <location href="repodata/cfdb36c3de1a5ca7d522db63bc2e12bc8c40025beb9cbe8079fca15222b85224-filelists.sqlite.bz2"/>
[root@tlielax ~]# ls -l /var/cache/yum/fedora-chromium/*primary.sqlite
-rw-r--r--. 1 root root 52224 Oct 10 09:12 /var/cache/yum/fedora-chromium/2edeb55260fe1e0fc9a00d6baf92bef74976f9393a82e7eb502c342a4bde2d38-primary.sqlite

...and here's the same for the fedora-debuginfo repo:

# egrep '(primary|filelists).sqlite' /var/cache/yum/fedora-debuginfo/repomd.xml 
  <location href="repodata/1e9f3fb809364d755fa4864aefebf1eaec9b10b0fd6dda330cbe6489075afce9-primary.sqlite.bz2"/>
  <location href="repodata/6ad3ddc0b0eb5b314f41c263f9625a9528ed699d8159bebdec56203ab736ccfe-filelists.sqlite.bz2"/>
[root@tlielax ~]# ls -l /var/cache/yum/fedora-debuginfo/*primary.sqlite-rw-r--r--. 1 root root 9783296 May 21 21:49 /var/cache/yum/fedora-debuginfo/1e9f3fb809364d755fa4864aefebf1eaec9b10b0fd6dda330cbe6489075afce9-primary.sqlite

...I'll leave it to you to tell me whether this is correct or not.

Maybe I'm just unlucky when I try to update my machine but this problem happens somewhat frequently, and it often takes quite a while for it to sort itself out (up to an hour in some cases).

It really doesn't make any sense to me to ever present a half-baked repo. Wouldn't it be better to sync the data to an alternate spot and then rename the files into place once everything is there?

Comment 3

Zdeněk Pavlas

2011-10-11 07:35:27 UTC

Right, fedora-chromium is ok.  FAIL, didn't read logs properly :/

> repomd.xml:
> <location href="repodata/1e9f3fb...-primary.sqlite.bz2"/>
> <location href="repodata/6ad3ddc...-filelists.sqlite.bz2"/>

> /var/cache/yum/fedora-debuginfo:
> 1e9f3fb8...-primary.sqlite

> failing url:
> ftp://... 6ad3ddc0b0eb5b314f4 ...

So the existing 'primary' file and the requested 'filelists' are referenced from the same repomd, that's what I wanted to check, thanks.

Now things make more sense..  There's likely nothing wrong with mirrors- it's just yum using stale 'repomd.xml'.  That should not happen right after 'yum clean all', maybe we've hit a bug there.

I'm pretty sure that removing '/var/cache/yum/fedora-debuginfo/repomd.xml' and running yum again would fix this issue.  Before you do that, could you please post 'ls -l' of that directory to see timestamps?  Thanks!

Comment 4

Jeff Layton

2011-10-11 10:15:43 UTC

Here you go. I've still not tried to update the machine, so let me know if you need more info.

$ ls -l /var/cache/yum/fedora-debuginfo/
total 9584
-rw-r--r--. 1 root root 9783296 May 21 21:49 1e9f3fb809364d755fa4864aefebf1eaec9b10b0fd6dda330cbe6489075afce9-primary.sqlite
-rw-r--r--. 1 root root       0 Nov 20  2010 395cf51348d27f5af6370e92908b5de9ac606390f589781007babc3c954658ec-filelists.sqlite.bz2
-rw-r--r--. 1 root root       0 Jun 14 15:40 6ad3ddc0b0eb5b314f41c263f9625a9528ed699d8159bebdec56203ab736ccfe-filelists.sqlite.bz2
-rw-r--r--. 1 root root       0 Oct 10 05:52 cachecookie
-rw-r--r--. 1 root root   17320 Oct 10 05:52 metalink.xml
drwxr-xr-x. 2 root root    4096 May 14 11:18 packages
-rw-r--r--. 1 root root    3131 May 18 09:36 repomd.xml

Comment 6

Matt Domsch

2011-10-11 15:45:36 UTC

5e6f5e16a... is the proper sha256sum for that repomd.xml file, and that is what all of the Fedora app servers are returning to that query, as far as I can tell.

However, the ftp.nsysu.edu.tw mirror clearly has the wrong repomd.xml.  Normally MirrorManager would recognize this and throw out the mirror.

However, I can't find the nsysu.edu.tw mirror in the MirrorManager database anywhere.  No URLs by that name, nothing obvious...  So I'm not sure how you'd be getting directed there.  Also note the URL pasted here points at development/15 which predates the release, and is in fact removed on the master mirrors.

Oh look...  i386/debug/repodata from ncsu is correct.

Comment 8

Jeff Layton

2011-10-11 17:05:15 UTC

I don't see anything in the logs that mentions the nsysu.edu.tw URL so I'm not sure that I ever did go there. It's possible I fetched the file from there a while back, but that seems unlikely -- I'm using yum-fastestmirror and I'm on east coast of US -- nowhere near .tw...

I'll hold off for a bit on updating the machine in case there's any other info under /var/cache/yum or that would interest you...

Comment 9

Matt Domsch

2011-10-11 17:12:07 UTC

Thanks Jeff.  That looks like a good metalink for the correct repomd.xml file.  You're still getting failures?

yum-fastestmirror could well have a bad cache itself - generally it isn't needed as much anymore because MM does a good job of handing out fast mirrors (statistically) preferred over slow mirrors.

Comment 10

Matt Domsch

2011-10-11 19:01:22 UTC

Chatting with Jeff via IRC, he had incorrect data in /var/cache/yum/fedora-debuginfo/ (particularly the repomd.xml file, with timestamp newer than what was on the Fedora mirrors).  yum clean all was not removing it.  By manually removing that directory, yum was able to download the correct repomd.xml file from a mirror, and everything else started working as expected.

Unclear why yum clean all doesn't just nuke /var/cache/yum/*, or at least nuke all the repomd.xml files previously downloaded.  Also unclear why it wasn't 
downloading new repomd.xml files.

In Jeff's cache:

-rw-r--r--. 1 root root 3131 May 18 09:36 /var/cache/yum/fedora-debuginfo/repomd.xml

On the mirrors:
-rw-rw-r--    1 ftp      ftp          3132 May 17 13:33 repomd.xml

Problem fixed for Jeff, so will close, but curious what the underlying failure really is.

Comment 11

Zdeněk Pavlas

2011-10-12 09:16:51 UTC

Matt, thanks for help!

'yum clean' issue:

Repository 'fedora-debug' is disabled by default, but the 'auto-update-debuginfo' plugin enables it.  'yum clean all' cleans only enabled repositories and does not call 'prereposetup' hook, so the plugin does not run beforehand.

FIX: Make 'yum clean' call prereposetup hook (but not actual reposetup, since we don't want to download metadata just to remove it)?

'repomd.xml' timestamp:

When urlgrabber downloads a file, it's timestamp is set to whatever was in HTTP header.  The 'cachecookie' file is used to check if metadata are current.  This file is touched even if download/verification of new 'repomd.xml' fails and yum reverts to the old one.  This should be changed too, probably.

Downloading an older repomd.xml file is fine (reget=None), but yum then calculates ts=max(timestamp) from <timestamp> tags in it, and refuses to use it if new_ts < old_ts.  Maybe we should turn off that timestamp check, too?

        if (self.timestamp_check and
            old_repo_XML.timestamp > self.repoXML.timestamp):
            logger.warning("Not using downloaded repomd.xml because it is "
                           "older than what we have:n"
                           "  Current   : %sn  Downloaded: %s" %

Comment 12

Marc Francesch

2011-12-26 15:33:19 UTC

Had the same issue. Thanks for the info.