Minutes for storage phone conference 6th Oct 2006 ================================================= Present: Edinburgh Greig (chair and minutes) Lancaster Matt, Brian RAL Tier-1 Derek Apologies: Jens, Mark Apologies to those who tried to call in at 10am this morning, we had not booked the phone conference after moving the day of the storage meeting. 0. Review of actions Decided to skip this until next week when more people will be attending. 1. DPM firewalling for rfio Did not discuss as Olivier was not in attendance. Essentially though you should firewall the dpm and dpns daemon ports. 2. Status of dCache and DPM GIP plugins Lancaster have not been able to deploy the `du` plugin yet but plan to use it once the current issues with their dCache have been resolved (see below). 3. Lancaster dCache problem Where to start? a) Missing dzero directory from PNFS After the meeting Matt reported that restarting PNFS caused the dzero directory to come back. However, it still it not clear what caused it to disappear in the first place. Matt should ask local dzero users to copy files into/out of the dCache to check that things are working. b) End of File errors when trying to copy files This is due to the max number of logins on the gridftp doors being reached which is caused by connections to the doors not being released by the dCache and therefore remaining the the CLOSE_WAIT state. This is caused by the dCache using blocking-IO in the code. The problem is know and should be fixed in the next release of dCache. IC-HEP have experienced the same problem, as have Edinburgh (just happened yesterday). It has probably been triggered by the large number of file transfers that have been taking place recently. The problem is resolved by restarting the gridftp doors (not ideal, I know) and the doors should be configured to allow a greater number of max logins. You can check if this is really the problem by trying to telnet to port 2811 of your gridftp door and looking for an end of file error. c) lcg-rep problems with atlas users Not had time to check this problem, but it seems similar to a problem they previously had with H1 users which appeared to be an permissions issue with the replica catalog they were using. Copying files using srmcp worked fine. d) Number of file replicas not decreasing Having reconfigured the replica manager to keep the number of replicas on the system between 1 and 1 (yes, 1) the dCache still has a large number of replica files in it. This isn't too surprising since dCache does tend to keep hold of files on disk until it _really_ has to remove them. Greig suggested copying in a large volume of dteam data to see if this would trigger the garbage collection. Derek suggested using the dCache command 'sweeper free ' in each of the pool cells. 'sweeper ls -l' returns information on each of the cached copies of files in each pool. Greig confirmed that this command works. Matt can script it to prevent him having to do everything by hand. Once cleaned up, Lancaster should be able to deploy the new GIP plugin which should provide more accurate accounting for them (since all of the file replicas will have been removed). 4. Supporting storage solutions Did not discuss. 5. AHM round up (if anyone who was at AHM attends) Brian gave a quick summary of storage related issues that came up during AHM. He mentioned the talk (from a GridPP person) about using http as a replacement for gridftp. The talk stated that they would only use single encrypted data stream, even in the case where multiple streams were being used to transmit data. He mentioned that a commercial research facility was looking at something similar. Brian also mentioned the use of SRB by the Oxford Grid project. At this time there is no need for us to consider SRB as it has not been requested as a baseline service by the LHC experiments. Only SRM is required. 6. GridPP17 Did not discuss. 7. AOB Jon Wakelin had mentioned that he would join the meeting today to report on his work with StoRM. Unfortunately he was unable to connect to the meeting due to the problems with the phone conference booking.