Accessing pagure's performances via mod_wsgi-express
Wednesday, March 2 2016
By Pierre-Yves on Wednesday, March 2 2016, 17:02 - Général
Tuesday, January 5 2016
By Pierre-Yves on Tuesday, January 5 2016, 18:59 - Général
Friday, December 11 2015
By Pierre-Yves on Friday, December 11 2015, 12:24 - Général
Every once in a while we make changes to dist-git in the Fedora infrastructure. This means, we need to test our changes to make sure they do not break (ideally, at all).
These days, we are working on adding namespacing to our git repos so that we can support delivering something else than rpms (the first use-case being, docker). So with the current set-up we have, we added namespacing to pkgdb which remains our main endpoint to manage who has access to which git repo (pkgdb being in a way a glorified interface to manage our gitolite). The next step there is to teach gitolite about this namespacing.
The idea is to move from:
/srv/git/repositories/<pkg1>.git /srv/git/repositories/<pkg2>.git /srv/git/repositories/<pkg3>.git /srv/git/repositories/<pkg4>.git
To something like:
/srv/git/repositories/rpms/<pkg1>.git /srv/git/repositories/rpms/<pkg2>.git /srv/git/repositories/rpms/<pkg3>.git /srv/git/repositories/rpms/<pkg4>.git /srv/git/repositories/docker/<pkg2>.git /srv/git/repositories/docker/<pkg5>.git
But, in order to keep things working with the current clone out there, we'll symlink the rpms namespace to one level higher in the hierarchy which should basically keep things running as they are currently.
So the question at hand is, now that we have adjusted our staging pkgdb and dist-git, how do we test that fedpkg still works.
This is a recipe from bochecha to make it easy to test fedpkg in staging while not breaking it for regular use.
It goes in three steps:
1. Edit the file /etc/rpkg/fedpkg.conf and add to it:
[fedpkgstg] lookaside = http://pkgs.stg.fedoraproject.org/repo/pkgs lookasidehash = md5 lookaside_cgi = https://pkgs.stg.fedoraproject.org/repo/pkgs/upload.cgi gitbaseurl = ssh://%(user)firstname.lastname@example.org/%(module)s anongiturl = git://pkgs.stg.fedoraproject.org/%(module)s tracbaseurl = https://%(user)s:%(password)email@example.com/rel-eng/login/xmlrpc branchre = f\d$|f\d\d$|el\d$|olpc\d$|master$ kojiconfig = /etc/koji.conf build_client = koji
2. Create a fedpkgstg (the name of the cli must be the same as the title of the section entered in the config file above)
sudo ln -s /usr/bin/fedpkg /usr/bin/fedpkgstg
3. call fedpkgstg to test staging and fedpkg to do your regular operation against the production instances
Thursday, November 19 2015
By Pierre-Yves on Thursday, November 19 2015, 15:22 - Général
I have recently been working on a new small project, an API to query the information stored in the meta-data present in the RPM repositories (Fedora's and EPEL's).
These meta-data include, package name, summary, description, epoch, version, release but also changelog, the list of all the files in a package. It also includes the dependencies information, the regular Provides, Requires, Obsoletes and Conflicts but also the new ones for soft-dependencies: Recommends, Suggests, Supplements and Enhances.
With this project, we are exposing all this information to everyone, in an easy way.
mdapi will check if the package asked is present in either of the updates-testing, updates or release repositories (in this order) and it will return the information found in the first repo where there is a match (and say so) So for example: https://apps.fedoraproject.org/mdapi/f23/pkg/guake?pretty=True*
shows the package information for guake in Fedora 23, where guake has been updated but the latest version is in updates not updates-testing. Therefore it says "repo": "updates".
Just to show you, here is the result of a little test performed with the apache benchmark tool:
$ ab -c 100 -n 1000 https://apps.fedoraproject.org/mdapi/f23/pkg/guake This is ApacheBench, Version 2.3 <$Revision: 1663405 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking apps.fedoraproject.org (be patient) Completed 100 requests Completed 200 requests Completed 300 requests Completed 400 requests Completed 500 requests Completed 600 requests Completed 700 requests Completed 800 requests Completed 900 requests Completed 1000 requests Finished 1000 requests Server Software: Python/3.4 Server Hostname: apps.fedoraproject.org Server Port: 443 SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES128-GCM-SHA256,4096,128 Document Path: /mdapi/f23/pkg/guake Document Length: 1843 bytes Concurrency Level: 100 Time taken for tests: 41.825 seconds Complete requests: 1000 Failed requests: 0 Total transferred: 2133965 bytes HTML transferred: 1843000 bytes Requests per second: 23.91 [#/sec] (mean) Time per request: 4182.511 [ms] (mean) Time per request: 41.825 [ms] (mean, across all concurrent requests) Transfer rate: 49.83 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 513 610 207.1 547 1898 Processing: 227 3356 623.2 3534 4025 Waiting: 227 3355 623.2 3533 4024 Total: 781 3966 553.2 4085 5377 Percentage of the requests served within a certain time (ms) 50% 4085 66% 4110 75% 4132 80% 4159 90% 4217 95% 4402 98% 4444 99% 4615 100% 5377 (longest request)
Time per request: 41.825 [ms] (mean, across all concurrent requests)
We are below 42ms so (0.042 second) to retrieve the info of a package in the updates repo and that's while executing 100 requests at the same time on a server that is in the US while I am in Europe.
- Note the ?pretty=True in the URL, this is something handy to view the JSON
returned but I advise against using it in your applications as it will increase the amount of data returned and thus slow things down.
Note2: Your mileage may vary when testing mdapi yourself, but it should remain pretty fast!
Wednesday, August 5 2015
By Pierre-Yves on Wednesday, August 5 2015, 11:02 - Général
Faitout is an application giving you full access to a postgresql database for 30 minutes.
This is really handy to run tests against.
For example, for some of my applications, I run the tests locally against a in-memory sqlite database (very fast) and when I push, the tests are ran on jenkins but this time using faitout (a little slower, but much closer to the production environment). This setup allows me to find early potential error in the code that sqlite does not trigger.
Faitout is running the cloud of the Fedora infrastructure and since this cloud has just been rebuilt, we had to move it. While doing so, faitout got a nice new address:
So if you are using it, don't forget to update your URL ;-)
See also: Previous blog posts about faitout
Thursday, July 23 2015
By Pierre-Yves on Thursday, July 23 2015, 08:43 - Général
flask is a micro-web-framework in python. I have been using it for different projects for a couple of years now and I am quite happy with it.
I have been using it for some of the applications ran by the Fedora Infrastructure. Some of these applications could be re-used outside Fedora and this is of course something I would like to encourage.
One of the problem currently is that all those apps are branded for Fedora, so re-using them elsewhere can become complicated, this can be solved by theming. Theming means adjusting two components: templates and static files (images, css...).
jinja2 the template engine in flask already supports loading templates from two different directories. This allows to ask the application to load your own template first and if it does not find them, then it looks for it in the directory of the default theme.
Code wise it could look like this:
# Use the templates # First we test the core templates directory # (contains stuff that users won't see) # Then we use the configured template directory import jinja2 templ_loaders =  templ_loaders.append(APP.jinja_loader) # First load the templates from the THEME_FOLDER defined in the configuration templ_loaders.append(jinja2.FileSystemLoader(os.path.join( APP.root_path, APP.template_folder, APP.config['THEME_FOLDER']))) # Then load the other templates from the `default` theme folder templ_loaders.append(jinja2.FileSystemLoader(os.path.join( APP.root_path, APP.template_folder, 'default'))) APP.jinja_loader = jinja2.ChoiceLoader(templ_loaders)
Adjusting static files
This is a little more tricky as static files are not templates and there is no logic in flask to allow overriding one or another depending on where it is located.
To solve this challenge, I wrote a small flask extension: flask-multistatic that basically allows flask to have the same behavior for static files as it does for templates.
Getting it to work is easy, at the top of your flask application do the imports:
import flask from flask_multistatic import MultiStaticFlask
And make your flask flask application multistatic
APP = flask.Flask(__name__) APP = MultiStaticFlask(APP)
You can then specify multiple folders where static files are located, for example:
APP.static_folder = [ os.path.join(APP.root_path, 'static', APP.config['THEME_FOLDER']), os.path.join(APP.root_path, 'static', 'default') ]
Note: The order the the folder matters, the last one should be the folder with all the usual files (ie: the default theme), the other ones are the folders for your specific theme(s).
Patrick Uiterwijk pointed to me that this method, although working is not ideal for production as it means that all the static files are served by the application instead of being served by the web-server. He therefore contributed an example apache configuration allowing to obtain the same behavior (override static files) but this time directly in apache!
So using flask-multistatic I will finally be able to make my apps entirely theme-able, allowing other projects to re-use them under their own brand.
Monday, June 29 2015
By Pierre-Yves on Monday, June 29 2015, 09:41 - Général
A while back I gathered some numbers about the number of participants to some election held in Fedora.
With the results of the new FESCo election being announced I wanted to go back and see the new trend:
FESCo (voters) 2008-07 150 2008-12 169 2009-06 308 2009-12 216 2010-05 180 2010-11 240 2011-06 200 2011-12 225 2012-06 236 2012-12 206 2013-06 166 2014-02 265 2014-07 195 2015-01 283 2015-06 90
As you can see, this last election was the one with the lowest number of participants since at least July 2008.
Friday, June 26 2015
By Pierre-Yves on Friday, June 26 2015, 07:37 - Général
I just wanted to point out a small feature added to pkgdb recently.
If you are dealing with someone who is slow to answer on bugs, irc or emails, it may give you an insight as to why that is.
Note: I am in no way saying that Paul is slow to answer bugs, irc or email, and have merely used him to illustrate my thoughts following up on his post about the Red Hat summit and I shall not be held responsible for any variations in Paul's response time :-)
Thursday, June 25 2015
By Pierre-Yves on Thursday, June 25 2015, 10:49 - Général
Recently I have been looking into Server-sent events, also known as SSE or eventsource.
The idea of server-sent events is to push notification to the browser, in a way it could be seen as a read-only web-socket (from the browser's view).
Implementing SSE is fairly easy code-wise, this article from html5rocks pretty much covers all the basics, but the principle is:
Server-side, things are also fairly easy but also need a little consideration:
- You need to create basically a streaming server, broadcasting messages as they occurs or whenever you want.
- The format is fairly simple:
data: <your data> \n\n
- You cannot run this server behind apache. The reason is simple, the browser keeps the connection open which means apache will keep the worker process running. So after opening a few pages, apache will reach its maximum number of worker processes running, thus ending up in a situation where it is waiting forever for an available worker process (ie: your apache server is not responding anymore).
So after running into the third point listed above, I moved the SSE server out of my flask application and into its own application, based on trollius (which is a backport of asyncio to python2), but any other async libraries would do (such as twisted or gevent).
After splitting the code out and testing it some more, I found that there is a limitation on the number of permanent connection a browser can make to the same domain. I found a couple of pages mentioning this issue, but the most useful resource for me was this old blog post from 2008: Roundup on Parallel Connections, which also provides the solution on how to go around this limitation: the limit is per domain, so if you set-up a bunch of CNAME sub-domain redirecting to the main domain, it will work for as many connection as you like :-) (note: this is also what github and facebook are using to implement web-socket support on as many tabs as you want).
The final step in this work is to not forget to set the HTTP Cross-Origin access control (CORS) policy in the response sent by your SSE server to control cross-site HTTP requests (which are known security risks).
So in the end, I went for the following architecture:
Two users are viewing the same page. One of them edits it (ie: sends a POST requests to the flask application), the web-application (here flask) processes the request as usual (changes something, updates the database...) and also queue a message in Redis information about the changes (and depending on what you want to do, specifying what has changed).
This way, the first user updated the page and the second user had the changes displayed automatically and without having to reload the page.
Wednesday, June 17 2015
By Pierre-Yves on Wednesday, June 17 2015, 10:04 - Général
How to get started with contributing to pkgdb2.
Thursday, May 7 2015
By Pierre-Yves on Thursday, May 7 2015, 12:26 - Général
A little while ago I presented a script allowing to search for the packages of a specified user and see which are missing from either anitya or are not being monitored in pkgdb2.
This script however, only check someone's packages and someone time we want to check a number of packages at once, eventually, all the packages matching a given template.
This new script does just that:
$ python pkgs_not_in_anitya_2.py 'drupal-*' drupal-service_links Monitor=False Anitya=False drupal-calendar Monitor=False Anitya=False drupal-cck Monitor=False Anitya=False drupal-date Monitor=False Anitya=False drupal-workspace Monitor=False Anitya=False drupal-views Monitor=False Anitya=False
If you are interested, feel free to use the script
Wednesday, May 6 2015
By Pierre-Yves on Wednesday, May 6 2015, 09:46 - Général
The election to select the talks for flock 2015 has started yesterday.
Anyone having signed the FPCA and being in one more group can participate to this election and help selecting the most interesting talks to be held at flock 2015 in Rochester (NY). Some of the talks submitted there look really interesting, I am looking forward seeing the agenda and I hope the ones I want to see will not conflict too much :-)
This year the election is using the simplified range voting approach. The principle is the same as for the classical range voting, but instead of having the possibility to score each candidate between 0 and X (X being the number of candidates, which is 132 for this election), you have the possibility to score each candidate between 0 and 3.
You can of course make your own scale but I went for something along the lines of:
- 0: not really interested by this talk
- 1: can be interesting, not sure
- 2: looks like an interesting talk
- 3: I really want to see this talk
And you, did you vote?
Monday, April 20 2015
By Pierre-Yves on Monday, April 20 2015, 19:01 - Général
This year, for the first time, I have been lucky enough that I could attend PyCon, the Python Conference.
This conference changes location every two years and this year was the second edition at the Palais des Congrès in Montréal, Canada.
Before I venture further into the conference itself, I would like to thank the organizers. The location was great! The organization flawless! And, as an attendee, everything went really smooth.
The conference itself is divided upon three sections
- The tutorials and the language summit (2 days)
- The conference per say (3 days)
- The sprints (4 days)
I did not attend the tutorials but I was invited to the language summit by Kushal Das (PSF board member and CPyton contributor). I was a unique occasion for me to meet and discover how things are discussed and decided within the python community. I must say it made me want to participate more in this community, join the mailing lists and, who knows, maybe try to tackle some easyfix bugs :)
During the summit we had a number of presentation about alternative python compilers like jython. We also had a short presentation by Guido about changes coming in python 3.5 to support declaring types in the function definition. Another interesting discussion was around the requests library and if it could ever make it into the standard library. While I think that specific question wasn't really answered during the summit, it triggered some interesting discussion around endorsing some external libraries within the documentation of the standard library (ie: advising users to use requests on the urllib documentation pages). Another really interesting topic that has been presented was the state of python on mobile platform (Windows Mobile, iOS and Android). While there are still some more work that needs to happen things seems to be progressing on that front and I'm quite looking forward the day we'll be able to easily ship python application in the different store.
The second day of the tutorials was more relax for me. I took this opportunity
to wander around Montreal a little and joined the crew of volunteers at the end
of the morning to help preparing the swag bags for every attendee of the
We first took out all the goodies shipped to the conference by the different sponsors and align them on two long tables. Then in the middle of the afternoon we started the 'bag stuffing' process :) This is a complicated process in which experts are carrying bags along the two long tables and another set of experts are translated items from the table into the bags.
Placing myself at the very beginning of the chain, I have probably been in contact with 2500+ bags of the 3000+ bags prepared (I would set the bags and be helped by one or two person that would either give away the bags or help me setting them up depending on the stash of prepared bags :)).
These were some interesting, fun, relaxing and sportive two and half hours! If you have not had the opportunity to stuff bags this year and are going to pycon next year, I highly recommend you to join this crew. It is a lot of fun!
The following three days have been the conference itself. To summarize, here is an overview of the talks I went to over the three days: Friday
- Opening: Julia Evans
- Keynote: Catherine Bracy
on the Coding For America project and in a broader sense what I would call, civic coding. (ie: how developper can help the community at large by making publicly accessible information and tools). This was a really great keynote, her talk was inspiring and motivating as well as calling for further reflection upon the roles of FOSS developper in our society within our field of expertise (developing) but outside our traditional scope (web, desktop, OS, company).
- Machine learning 101: Kyle Kastner
This was also a very interesting talk going over the different machine learning algorithm, libraries and use-cases. That helped getting an overview of the field
- Introduction to HTTPS: A comedy of errors: Ashwini Oruganti & Christopher Amstrong
This was presenting what are the current issue when dealing with https in general, within python or not. I can't say I learned new things in there but it is always good to get refreshed on this topic
- Insite the Hat: Python @ Walt-Disney Animation Studios: Paul Hilderbrandt
This was a really interesting presentation about the use of IT in general (and python in particular) at Walt-Disney Animation Studios and of course it was full of pretty pictures from Big Hero 6 as well as some other pretty pictures from a couple of other movies. Paul also presented the overview of how animation movies are made and how Disney developed their own tools to facilitate this process insisting on the idea that the tools have to adapt to the artist rather than the other way around.
- How to interpret your own genome using (mostly) python: Titus Brown
This talk presented tools and workflow that can be used to analyze and compare genomes, taking a population that had a particular history as example and going down into the genome to figure out what (at the gene level) makes this population so specific. It also gave an overview of the possibility for high-throughput genome sequencing and the application that can derive from it as well as touching the surface of the ethical concerns that raises from these technologies.
- How to build a brain with Python: Tevor Bekolay
While still being bioinformatics this was a very different topic than the previous talk I attended. This presentation was really about the inner (ie: chemical and physical) modeling of neurons of a brain. The presentation started by introducing a couple of application used to model a single neuron and then introduce their own application used to model several neurons at once. Quite impressive and interesting presentation although knowing more about the biochemical and biophysical properties of the brain would have probably lead to a better comprehension of the work presented :)
- Introducing python wats: Amy Hanlon
While I must say I knew most the example she presented of curious behavior of python, I must say that I did not know completely the reason of these behaviors. The presentation was really nice in that it gave some clues and as well as some tools to help figuring out what is actually happening in the code and why these, sometime surprising, behaviors.
- Learning from other's mistakes: Data-driven analysis of python code: Andreas Dewes
This was an interesting presentation describing the approach develop by this company to do static code analysis but considering the code not as text but as a graph. This approach allows to find out bugs in the code due to, for example, typos in property names. It seems that the service is freely available but unfortunately, if I understood correctly, the tool is not FOSS.
- Technical Debt - the code monster in everyone's closet: Nina Zakharenko
The interesting bit about this presentation is that anyone that worked on a reasonable size project could relate to what was presented. There are many times where I thought that I have been in the situation described and some time when I thought I wasn't doing too bad (but here I guess it depends on the projects). There were some good elements to help figuring out the size of the debt as well as some good ideas on how to organize the work to reduce this debt.
- Achieving continuous delivery: An automation story: James Cammarata
This presentation was about Ansible and how different companies are using it to automate their deployments. Several examples of companies were given, some even integrating Ansible with an IRC bot allowing everyone on the IRC channel to see what the other admins are doing.
- Build and test wheel packages on Linux, OSX and Windows: Olivier Grisel
Wheel are a format that can be used to compile python packages into binaries that can then be installed on multiple platforms. There are clearly some advantages in this but I am not quite convince especially with regards to architecture specific code and the different architectures that we have today (x86, arch, arch64, ppc...) But anyway, since Fedora does not allow shipping binary files directly wheel isn't quite an option for us. On the other hand it might be one for applications such as liveusb-creator or pyrasite that aim at being cross-platform.
- Graph database patterns in python: Elizabeth Ramirez
The presenter of this talk works at the New-York Times journal and was presented the approach the use internally (as well as the tools and library) to store semantic concepts, link them and navigate the graph they make. After the presentation I ended up having a very interesting talk about the difference between full-graph database and rdf databases and what the former allows that the later does not. While I am still a little unclear about this difference, it was a really interesting conversation and something I would like to look further into if I was still working with/on semantic web technologies.
- Keynote: Van Lindberg
This was a presentation from the head of the PSF board about the state of the python community and python in general, how it went from being a trendy language when it was created into something stable and sure these days, but also how other languages are growing, potentially threatening python by being the new trendy languages. Community wise, I have written one quote from this talk that I really like:
A community where people interact only when they are paid to do so is not a community, it's a bunch of mercenary
- keynote: Jacob Kaplan-Moss
This was a great talk about the perception that we have as developers of themselves. For example, did you realize that there are two kinds of developers: the great ones and the terrible ones while if the quality of a developers could be quantified we know that just like everything else it would follow a normal distribution, ie: most people would be average developers and only a few would be great and a few would be terrible. If you have seen it I would like to say:
Hello, I'm pingou, I'm a mediocre programmer
If you haven't seen it, I invite you to watch it as it was an inspiring talk, really.
- Interactive data for the web - Bokeh for web developers: Sarah Bird
Bokeh is a library that can be used to create interactive graph that can be included in web pages. The examples shown during the presentation were really impressive and while it probably needs some understanding of the different ideas, concepts and of the library itself, it is definitively something I will look into the next time I have to do some data visualization.
- WebSockets from the wire up: Christine Spang
While I have heard about web-socket I have not had the opportunity to play with them more than this. In this talk the history and principles of web-socket was described, giving a nice idea of what they can be used for. I must say I know kinda want to play more with them, build more reactive UI using web-sockets. However, for the projects I work on these days I feel it would be a little bit overkill. Maybe for next one ;-)
- Type hints: Guido Van Rossum
This was a very similar presentation to the one Guido gave during the language summit, presenting the work coming in python 3.5 to support type documentation in function definition. Here, as well as during the language summit, I got quite enthusiast about this idea but the syntax of putting the type in the function definition is really not appealing to me. It makes the function definition both harder to read and, in some case, much longer. To be honest I would love to see the same syntax be supported in docstring which is where I believe it belongs (plus, as a bonus, it kind of encourage developers to document their code, if you start writing docstring for the type, maybe you can add documentation about the arguments themselves and the function, and so on).
- Keynote: Gary Bernhardt
This keynote was probably the most technical keynote we had (except for Guido's presentation just before). It presented a comparison between strong type languages and dynamic type languages.
This is it for the talks I attended. There are more talks I would have liked to see but either I was doing something else or there was another talk at the same time. Luckily all the talks have been recorded and are available on youtube.
Among the talks I would like to see are:
- Building secure systems - lvh
- What can programmers learn from pilots? - Andrew Godwin
- "Words, words, words": reading Shakespeare with Python - Adam Palay
- Is your REST API RESTful - Miguel Grinberg
- l18n: World domination the Easy Way - Sarina Canelake
- Good test, Bad test - Dan Crosta
- How our engineering environments are killing diversity (and how we can fix it). - Kate Heddleston
- Open Source for Newcomers and the People who want to welcome them - Shauna Gordon-McKeon
- Cutting off the internet: Testing applications that uses requests - Ian Cordasco
- Rethinking packaging, development and deployment - Domen Kozar
- Describing descriptors - Laura Rupprecht
- Avoiding burnout, and other essentials of Open Source self care - Katheleen Danielson
- Python performance profiling: the Guts and the Glory - A. Jesse Jiryu Davis
As you can see I'm in to spend few hours watching youtube videos :)
The third part of the conference was the sprints.
The idea of the sprints is to take advantage of the fact that many developers
come to the conference to keep them a little longer and offer them projects to
During these four days, you can see people hacking on Django, MailMan, Jython, CPython itself, sage, pypy and many more projects. I took this opportunity to spend more time with the people from my team not that we don't work together most of the time but it is nice to be working together in the same room. As for the project, most of the time has been spent on making pagure closer to something we would want to deploy/use. I must say that at the end of this week, since are looking good. Pagure now has support for webhooks, pull-requests can be assigned, they have a score and the project can require a certain score for a pull-request to be merged. Basically, for what I want pagure needs: a) more documentation, b) more unit-tests and c) more tests and d) support to upload tarball/release (although this might arrive only in 0.2). So once documentation and unit-tests are there, I will tag a 0.1 release and move pagure to production (I'll announce it here so keep in touch! ;-))
As final words, I started this (long, sorry) blog post with saying how lucky I am to actually having been able to attend this conference and I would like to thanks Red Hat in general and more precisely the OSAS team that funded my flights and pass for the conference.
Tuesday, April 14 2015
By Pierre-Yves on Tuesday, April 14 2015, 23:16 - Général
Just wanted to make a quick note here.
Just heads up to those looking for a FOSS emoji database/project and big thanks to the developers and artists behind this awesome project!
Friday, April 3 2015
By Pierre-Yves on Friday, April 3 2015, 16:22 - Général
One of the earliest feature request of pkgdb2 (that was present in pkgdb1) is the browser search integration.
This integration is based on the OpenSearch specifications and basically allows to use pkgdb as one of the search engine of your web browser just like you can use google, duckduckgo or wikipedia.
I recently found out this feature is not so well known, so I thought I would present it and explain how to set it up (screenshot are on Firefox).
1/ Go to https://admin.fedoraproject.org/pkgdb and click on the list of search engines at the top right.
2/ Select the entry Add "Fedora PkgDB2: Packages"
That's it you are done for the most important step :)
Now something which I do and find most useful is:
3/ Go to Manage Search Engines...
There, with the search engine pkgdb packages associate the keyword pkgdb
Now, you can use your url bar as usual but when you enter pkgdb <something> it will search this <something> in pkgdb directly. So for example, if you want to search for guake in pkgdb, you would type in your url bar pkgdb guake.
The bonus point is that since there is only one package with this name, you will be immediately redirected to its page.
This way, when you want to quickly find information about a package in pkgdb, you can get it from your browser in one simple step (eventually two if several package match the keyword you entered).
Final bonus point? To access pkgdb directly, enter in the url bar: "pkgdb " (with a space at the end), without a keyword, Firefox will bring you directly to the front page of the application.
Wednesday, March 25 2015
By Pierre-Yves on Wednesday, March 25 2015, 09:57 - Général
You may have heard of a little pet project I have been working on recently, I called it progit but there already a more well-known project named progit (the pro git book).
So, after long deliberations, we decided to rename the project: pagure.
What is Pagure?
Pagure is a small git-centered forge project. You can host your code, your documentation, your tickets and have people contribute to the project by forking it and opening pull-requests.
All the information about the project is hosted in different git repositories, the code of course, but also the documentation as well as the metadata (discussion) of tickets and pull-requests. The idea being that one could host a project in multiples instances of pagure and keep them in sync.
What about the name?
Pagure is the generic (French) name for animals of the Paguroidea family which includes the well known Pagurus bernhardus. This little crab moves from shell to shell as it grows up. I found it was a nice analogy with this forge where project can move from place to place.
Where can I see it?
Pagure is still under development and pretty much changes every day. However, you can already see it, test it and poke at it via the dev instance we have running.
As you will see, pagure itself is being developed there, so feel free to open a ticket if pagure does not do something you would like (or does something you do not like).
Tuesday, March 24 2015
By Pierre-Yves on Tuesday, March 24 2015, 14:35 - Général
A little while ago, I blogged about the new package and new branch request processes.
These changes have been pushed to production yesterday.
What does this change for you, packager?
From there admins will review the package review on bugzilla and create the package in pkgdb (or refuse with an explanation).
If your package is already in Fedora, you can now directly request a new branch in pkgdb. Here there are multiple options
- You have approveacls on the package (thus you are a package admin) and the request is regarding a new Fedora branch: The branch will be created automatically
- You have approveacls on the package (thus you are a package admin) and the request is regarding a new EPEL branch: The request will be submitted to the pkgdb admins who will process it in their next run
- You do not have approveacls on the package, then your request will be marked as: `Pending`, this means that the admins of the package have one week to react. They can either approve your request and by setting it to Awaiting Review, or they can decline the request (for which they must specify a reason). After this one week (or sooner if the package admin set the request to Awaiting Review) the pkgdb admin will process the request like they do with the other.
Note: Even with this new workflow, requests are still manually reviewed, so the requests will not necessarily be processed faster (but if it is easier for the admins, they may run it more often!).
What does this change for you, admins?
Hopefully, the process will be much simpler for you. In short
- no need to log onto any system, you can do everything from your own machine and it should work out of the box
- much more automated testing (including checking if a package is present in RHEL and on which arch for EPEL requests)
- one tool to process the requests: pkgdb-admin distributed as part of packagedb-cli (aka: pkgdb-cli)
I hope this process makes sense to you and will make your life easier.
You are welcome to already use these processes, just let us know if you run into some problems, but for the time being both the old and the new processes are supported :-)
Wednesday, February 25 2015
By Pierre-Yves on Wednesday, February 25 2015, 13:21 - Général
The question was asked on the devel list earlier if there was a way to check all one's packages for their status in pkgdb and whether they are in anitya.
So I just cooked up quickly a small script to do just that, it retrieves all the packages in pkgdb that you are point of contact or co-maintainer and tells you if its monitoring flag is on or off in pkgdb and if it could be found in anitya.
For example for me (partial output):
$ python pkgs_not_in_anitya.py pingou * point of contact R-ALL Monitor=False Anitya=False R-AnnotationDbi Monitor=False Anitya=False ... guake Monitor=True Anitya=True igraph Monitor=False Anitya=False jdependency Monitor=True Anitya=True libdivecomputer Monitor=True Anitya=True metamorphose2 Monitor=False Anitya=False packagedb-cli Monitor=False Anitya=False ... * co-maintained R-qtl Monitor=False Anitya=False fedora-review Monitor=True Anitya=True geany Monitor=True Anitya=True geany-plugins Monitor=True Anitya=True homebank Monitor=True Anitya=True libfprint Monitor=True Anitya=True ...
If you are interested, feel free to use the script
By Pierre-Yves on Wednesday, February 25 2015, 10:17 - Général
So here is a summary of the situation:
A project X on sourceforge.net, for example with a homepage sourceforge.net/projects/X, releases multiples tarball named, X-1.2.tar.gz, libX-0.3.tar.gz and libY-2.0.tar.gz.
So how to model this.
The original approach taken was: the project is named X, so in anitya we should name it X and then the sourceforge backend in anitya allows to specify a Sourceforge project allowing to search X, libX or libY in the rss feed of the X project on SourceForge. Problem: when adding libX or libY on anitya, the project and homepage are all X and sourceforge.net/projects/X, while this is actually used to make project uniques in anitya (in other words, adding libX and libY won't be allowed).
So this is the current situation and as you can see, it has problems (which explains the two issues reported).
What are the potential solutions?
1/ Extend the unique constraint
We could include the tarball name to search for in the unique constraint, which would then change from: name+homepage to name+homepage+tarball
2/ Invert the use of name and tarball
Instead of having the project name be X with a tarball name libX, we could make the project be libX and the tarball be X.
This sounds quite nice and easy, but looking at the project currently in anitya's database, I found projects like:
name | homepage | tarball + + linuxwacom | http://sf.net/projects/linuxwacom/ | xf86-input-wacom brutalchess (alpha) | http://sourceforge.net/p/brutalchess | brutalchess-alpha chemical-mime | http://sourceforge.net/projects/chemical-mime | chemical-mime-data
So for these, the tarball name would become the project name and they would be pretty ugly.
I am not quite sure what is the best approach for this.
What do you think?
Thursday, January 22 2015
By Pierre-Yves on Thursday, January 22 2015, 10:52 - Général
A little while ago I blogged about a new process to request a new branch on an existing package.
The code to support this change is now under review but I thought I should document the workflow a little bit, so here is how I tried to design things:
Ideally, when the branch is approved and created in pkgdb by the admin, pkgdb will send a message on fedmsg, message that will be seen by a fedmsg-consummer that will automatically update the git repos within, say 2 minutes. That last part if almost ready and hopefully will be running soon.
« previous entries - page 1 of 11